i mean, it's not going to be detrimental (or is it?
)... but think about your food web for a second. what do worms eat? the bacteria and fungi that break things down in your soil... which is a good thing because it will release available nutrients for the plant... but 500 worms in 20 gal... that population will never survive entirely unless you're basically turning your planter into a worm farm because there just won't be enough food source for them... and some will end up starving to death. So you will have some die off... the population will self regulate. but in the mean time, if the worms gobble up as much bacteria and fungi as they can then who is going to be breaking down those foodstocks for the plant? another concern i have, worms have a tendency to neutralize things... like your acidic peat... and will basically turn your whole pot into compost. I can say that over time i've diluted my mix with too much compost, and that's why I lost my acidity... leading to less than ideal yields (but still healthy plants).
So overall... I'll be interested to see how this plays out if you're willing to leave them all in there; just seems like mega overkill to me and a sure way to disrupt the balance of the ecosystem. I mean, this is just me speculating on a few things and is by no means a definitive thought process... but remember...
less is more. start small, and go from there. you can
always add more if need be.