Vermiculite just isn't all that good for soil mixes. It does mess with the pH, but it's biggest sin is, it just flat retains too much water. Not a good trait for a aeration amendment.
No experience with pumice for the very reason you posted, $32 for 5 gallons vs $20 for 30 gallons made it a...