How much should I worry about hip pain?

Depends. There are many factors an associated symptoms that will tell us how much this needs looke into and how to treat it. So please see your doctor.