Typically, the plant removes those by itself. Those leaves will exhaust all their stored nutrients, turn yellow and then wither and fall off, or be easily plucked off with the slightest effort when they are done.
I'm in week 6 of Flower and I removed some of the Fan leaves that where blocking alot of the Bud sites that weren't getting light. There's still alot of leaves left though.
One of my cakes is pretty tiny and bushy and I plucked away some stuff later on to let it breathe.
But in general if it's not really neccessary it's not really neccessary, I pretty much left the other ones alone!