Please explain why does one of my eyes go inwards when i don't wear my glasses?

Your eyeglasses may: have a prism in it to keep your eye from going inward. Or you may just have a condition in which when you're tired, your eye wants to go inward, known as an esophoria. So that's nothing to be alarmed with, as long as you're wearing your glasses that help correct it. If its a new problem, see your eye doctor immediately, so s/he can determine the cause.

Refractive strabismu: You were far sighted as a child and there is a reflex to see at near which allows the lens to focus and at the same time turns the eyes inwards. I am guessing you were patched and have maintained good vision in each eye. When you wear the glasses, that takes some of the inturning energy away and the eyes stay straight. When the glasses are removed, the inturning becomes dominant.