How much should I worry about dizziness?

Depends. So many things can cause dizziness, we don't have the space and it would be a disservice to you for me to guess, particularly since I don't know many details that would help me help you. See a physician please.