Happens to me if I knock a plant on the floor or something.
When it happens there is just no getting the dirt off the buds, they are as sticky as flypaper. You can't blow it off, vacuum it off, or anything else. I thought of trying to freeze the buds and knock it off, but then that can just knock the trichomes off too.
Smoke it dirty or make hash.