not at all, many people actually remove a good many of those. I don't, but many growers do, search "defoliation for higher yield" although in my opinion,.... defoliating for higher yield is like bombing for peace or fucking for abstinence,lol
Your plant is just fine and probably didnt miss a beat.