What would the long term problem if I don't get my cataract removed from my eye be?

Decrease in vision. Generally cataracts cause reversible vision loss. As long as the cataracts are not contributing to glaucoma or inflammation( phacomorphic uveitis), one can just follow. However, waiting too long can make the surgery more difficult hence timely removal is preferred.