I understand your take on the 'it's human nature' part. And honestly, we'll never know if the tables were turned, would Africans or Asians (or any non-white entity) bedoing the same.
But since we're hypothesizing, here's my take.
Based on the existence of a particular culture or civilization, America/Britain (or whichever Western Colonist that applies), they are no more than a few centuries old.
Whereas civilizations like Egypt, India, and China,, have been around millenniums (each). I'm not even going to bring up that we may all have come from Africa based on genome studies.
But yet, by your logic, these countries civilizations turned out to be a little less colonial than it should?