yes, but they have rocks and stones.. this helps soil in fields from compacting and retain moisture. They also have worms that aerate the soil and provide additional value to the soil in the form of their castings. You will also note that farmers spend considerable time and money to prep their soil by installing weeping tiles through their fields in clay soil environments, and irrigation in sandy fields.
I wouldn't suggest throwing rocks into your pots, but if you can get a hold of lava rocks (the red porous ones) throw them in a burlap bag and beat them down with a hammer.. wash them well.. and mix those in- instead of perlite.. will achieve the same effect.
Sand drains very quickly and from what I understand washes out fairly easily, although I haven't used sand as an additive to anything.. I have heard of people putting sand on top of their soil because of bugs... fungus gnats I think? But may work well if you can't find anything else.