No reason to keep him unless you are trying to breed plants instead of growing weed.
Not to worry though, marijuana plants are very flexible. Your yield is much more based on the size of your lights over the number of plants you have. However many females you finally end up with, they tend to expand to make use of all the light. So five plants vs ten plants isn't 50% less. More like 5%-15% less - depending when you dump your males. Your remaining plants will just grow more to fit the light available.