It all comes down to personal preference and timing.
If you don't mind smoking sugar leaves, then only remove the fan leaves at harvest time and your all set.
If not, you should remove the sugar leaves at harvest time whenever you are in a location where humidity is high and the temps are low and also if you don't have too much to harvest at the same time.
Ideally, the best quality and best looking buds will be obtained by doing a dry trim. I will slow down the drying time thus preserving most of the terpenes!