Is it normal to have upper abdominal pain when you're constipated?

No. Not usually. Upper abdominal pain may be secondary to gastric or duodenal ulcer, gall stones or gall bladder disease, or pancreatic problems.
See answer. its not normal to have pain! As for constipation, if it is "backed up" enough to the level of the mid-colon then it can cause upper abdominal pain. Other causes of pain should be considered, depending on the location of pain (left upper pain or mid upper can be stomach inflammation/ulcer, while right upper pain can be gallbladder). If pain gets better with defecation, probably constipation.