You "get them outa there" before the pollen that they produce pollinates your other females resulting in seedy buds.
Pollen around plants doesn't change plants into males, it just means you either have males or hermies in your grow that WILL pollinate your females, they won't CHANGE any other...