You can put them outside any time, but you have to do it correctly. You can bring them out in the shade for a couple of days, then partial sun, and finally full sun over a few days..
As long as you do that, they will get used to the sun at nearly any stage of growth.