I have sudden vision loss very frequently. Should I be worried?

Maybe. I am suspicious that this may be migraine related. Go to an eye doctor for further evaluation.
Yes. Sudden vision loss (amaurosis fugax) is a symptoms which should never be ignored. Get checked out right away and be sure to tell your doctor when this occurs, for how long and what other symptoms you may be experiencing. This is not normal and should not be ignored.