i had the same issue, mine is much better now ive included the below:
Insulated ducting
built cardboard box around fan, seal gaps with duct tape
duct tape all connections
add a silencer to the end of your fan
I can run my fan at 100% now and can barly hear anything, the main noise issue now...