You are exhausting your tent to a fresh air intake? That would be my guess as to why you are smelling it in the house. Just because the duct connects to outside doesnt mean the air flows that way. Seems like by design its going to pull air in. So even if you close the duct off so you no longer use it for fresh air, wont it be fighting you the whole time and negating your exhaust?
Try this. If you disconnect the fan from ducting, and hold your hand up or put a piece of paper in front, is air blowing thru it?
Im guessing you just need to find a different place to exhaust to, and that your fan and filter will probably be fine as is.