Wondering if it's normal to have trouble seeing out my lazy eye?

Usually. A drifting eye is one that most likely does not see well and has not done so since childhood. Since it does not have good vision to keep it in alignment with the normal other eye, it will drift at times and in some all of the time. It is common to have this discovered at a later age than childhood if the condition was not discovered below age 8 when it might have been remedied.