Slow Roll, no! I'm 100% sure that stuff is male stuff, nanners! Still, as Kingrow1 said, it won't hurt that much, remove any male flowers you find. What he said about hermies is absolutely true as well, true hermies are really rare!
In your case, it sounds like late-flowering nanners! How come...