I was diagnosed with an adrenal tumor. My doctor says it's benign, but I'm having a lot of pain. Should I be worried?

Location. Most adrenal nodules are of benign nature and remain stable and non functioning, however they should be monitored on an interval based on size and location. Although you did not describe the nature and location of pain, you will need to discuss this further with your physician. Adrenal neoplasms can be dynamic and there is many cases reports of non secretory nodules becoming hornonally active.
???cause of pain??? Most non-functioning adrenal tumors that are less than 4 CM in size can be safely watched due to the low probability that they are malignant. Assuming that your tumor is that small, it would be highly unusual for the tumor to be causing any pain. I recommend that you seek out your physician's opinion what may be causing your pain.