I grow outside now, right next to the shed. I've used the shed as a darkroom for like 4 years. And the shed has been in the same location for over a decade. I even let the police look over the fence a few weeks ago when someone broke into a neighbors house. Nothing could be seen so I thought it would be easier to let them look than to make them suspicious. I've never had a problem with the police, I look like a soccer mom (which really does help). And there aren't people running into and out of my house.
But I would love to know the other reasons it is bad. I know that it would be hard to cool and heat, but I've taken care of that.