As a result, this may cause the eye to drift more. As you said, wearing your glasses or contact lenses more, will reduce the drifting.
Depending on how much the eye is drifting and if the eye remains in a deviated position, you may consider surgical correction where eye muscle surgery will align the eyes. Also, if you see a change in the position of your eyes over time, i encourage you to see an ophthalmologist (eye physician & surgeon) who will assess your condition, make sure it is not something bad causing the change in the eye position, and recommend a plan of action.
When dealing with this problem, you have several choices:
1) do nothing;
2) wear your glasses/contacts more;
3) if the double vision
bothers you, wear an eye patch;
4) if bothersome enough, seek an evaluation for surgery.
I hope this information helps.