I have to wear glasses all the time, will this make my vision worse?

No. The eyes will not change for better or worse because you correct the vision with glasses. So not wearing them will have no effect except to force you to see things blurry all the time. Wearing them will clarify the vision but not alter the basic power.