IMHO, your the caregiver, so the grow is legally (cough..cough) yours in your states eyes. If you have your grow locked and secured not allowing him access to the room or any keys or your product, that should help of the concern of, "he could access your grow" comes up.
I would ask the people who are running the medical caregiver system for exact advice. They would know the true answer, not the police or lawyers as these laws are too new for everyone to understand yet.
DISCLAIMER: I am not in your state, just tossing in my 2 cents worth for what it is worth.