Can glasses make your eyes even worse when you take them off?

No. However, the glasses make you see better without any focusing effort your internal lens. Your eye relaxes because the glasses are doing the work, and sometimes when you take them off, things "appear" more blurry.