Alright Bros, I figured out a pretty sweet solution after racking my brain for awhile. I took the cardboard box my fan came in and laid it horizontally. It was wide enough that my blower could still stand up straight inside (but any cardboard box will do as long as it fits in the next step). I then cut 6 inch holes for ducting in the cardboard in prep for the next step.
I then placed the box, with fan, inside one of the spare rubbermaid (think they're 25 gallon) storage containers I had. I then cut the same size 6 inch holes in the rubbermaid. Then, between the cardboard and the rubbermaid, I put spare egg carton shipping material for noise dampening (popcorn styro, crumpled newspaper, etc would work here).
I then connected the ducting through the outer and inner holes and put the lid on. Once the ducting is run out, it's about 80% more quiet than before. In fact, with the door closed to the grow room, it's almost silent from outside. Also, the exhaust is very quiet, just sounds like a faint dryer vent.
I'll post some pics tomorrow if someone hits me up with a PM reminder.