i've been thinking about this same idea for a few weeks now, as an energy saver and some natural-ness for my babies. but i decided against it for a few reasons. i'm assuming you have a grow room set up for your plants:
1) you can get pests on your plants when they are outside and transfer them to your room where they are given an ideal habitat to reproduce and wreak havoc on your babies when they get to the critical bud stage.
2) i had some lavender that i kept inside over the winter on my window ledge and when the warm weather came i started to put it outside again. a few days after, i noticed some other small plants shooting up in the soil around the lavender. so when it comes to my babies, i don't want anything competing with them. especially since you don't know what could be growing in there along side them, they could be pest attractants. you never know.
3) i shy away from anything that may attract people to my grow. such as putting my plants outside in the way of prying eyes. if you already have a set up inside, i would just stick with that. you can control the plant's environment better.
4) temp swings, sudden storms when you aren't home to bring the plants in, hail (depending on where you live).
but that's just my opinion.