Does not wearing your glasses make your vision worse?

Yes. Your eyes will continue to change and adjust in order to maintain your vision, sometimes even if 1 eye gets lazier.
NO. Not wearing glasses-if you need them---will not make your eyes worse, but you won't see as well as you should and this may cause or headache or eye strain--but no permanent damage will occur.