I did get the residue as well, but I really did not mind it. It was not visible on the flowers when I harvested, and it doesn't matter if it's on the leaves that are thrown away anyway.
I've found a scientific survey of the available measures, and surprisingly the milk application was one of the best methods available. It is not even known how exactly it works. The milk should be applied in full sunlight, because apparently the intense light transform something in the milk into a substance that is anti-fungal.