The simple fact that people are not property. Personal property is fine. Private property requires force.
People are not property in the sense that they could be "owned" by others against their will. I'm good with that.
However people are "property" in the sense that they can own themselves. In that sense it would be wrong for me or you to prevent people from using themselves in ways that they determine as long as that use does not infringe on others right to do the same.
Private property does not require force in every instance. For instance, you live in the woods and make a stick into a spear. You now "own" the product of your combined labor and a natural resource, no?
Removing people from occupied land requires force if the people being removed were there first, then the removing party is the aggressor.