The three things that made the biggest difference in sound dampening for me were as follows:
#1- i built a box for my exhaust fan using 3/4" MDF board, i built it so the the fan sits snugly between the boards when sealed, this does mean having to connect ducting before sealing the box of course, before sealing it i also packed the box full of foam packing peanuts, once sealed i caulked the small openings where the duct and power cord go through.
#2- i insulated my ducting, if your ducting isnt insulated then alot of the noise you hear isnt even the fan itself but rather its the "whooshing" air traveling through the ductwork, duct insulation is cheap, dont buy the pre-insulated stuff though because the inner ducting on it is actually plastic and isnt suitable for exhausting hot air, especially if your using HID lighting, you can buy just the insulation sleeves for cheap and pull it over your existing metal ducting.
#3- THIS MADE THE BIGGEST DIFFERENCE!- if youre using a speed controller on your fan then your probably using one of the cheap 15-20$ rheostat-style controllers, you need to upgrade to a "variac transformer" the reason for this is the cheap style controllers work by switching your fan on and off very quickly, this means the motor is still working at full power but just being constantly turned on/off to reach the desired speed you have set, this causes excessive wear and tear on your motor- shortening the lifespan of your fan, and is the true culprit behind atleast 50% of your fan noise, a variac transformer actually allows you to adjust the voltage being sent to the plugged in device, in this case the exhaust fan, which is the proper way to control fan speed, i sleep in the same room my tents are in so trying to go to sleep with all the noise was a constant struggle for me, after upgrading to variac transformers vs rheostat-style controllers i fall asleep with no issues!