Why Doesn't Mass Immigration Count as a Form of Colonialism?

One of the major problems we have in talking about immigration is that people tend to think about it in a way that obscures where the actual disagreement lies, and I think asking that question is a good way to get into the problem.

The social justice left has decided that there are certain times when a group of people can move from one community to another community and this is totally acceptable. They approve of these cases of relocation and call them “immigration,” defending them at every turn. However, there are times when the left says it is absolutely not ok for a group of people to move from one community to another community. These forms of relocation the left disapproves of are called “colonialism” (when they occur on an international scale) and “gentrification” (when they occur at the level of neighborhoods and cities).

The puzzle before us is simple: on what basis does the left declare some sorts of mass relocation to be legitimate and other forms to be illegitimate?