Thanks so much all for the responses!
@slowbus, Joe Macclennan has the right idea. Even with the fan off, the air pressure in the room is enough to push air out through the vent. I have been looking at dampers, but the way I figure, unless a damper is rated to open only when say, 450 cfm is...