You don't get gnats from having wet soil. Letting your soil dry out excessively probably wont solve the problem, either.
Gnats come from gnat pupa, which come from gnat larvae that come from gnat eggs laid by gnat females.
Excessive moisture gives them better conditions (and fungus too, which they eat), but I've seen swarms of gnats hovering and laying their eggs in apparently bone dry soil that's been left sitting out in the open. You have to water your plants anyways; it is best to take a few preventative measures which can also be used to achieve control of an infestation. Yellow sticky traps should be incorporated into every grow room (especially soil). I like the SAFER brand sticks that go right in the soil.
jondamon suggested other viable controls, most of those I believe are biological. Gnatrol is Bti, a strain of bacillus bacterium that is toxic to a few select species of insect larvae. If you can't find Gnatrol, you should be able to acquire some Mosquito Dunks (same ingredient), as they are sold in most Lowes/Home Depot type stores.
You can also vacuum around the pots each day to reduce the numbers of adults.