Should I see a doctor about a rash on my baby’s abdomen?

Yes. In general if the rash bothers your baby it should be addressed. There are many causes of rash from dry skin, to irritation from soap or detergent, to infection with virus, bacteria, or fungus/yeast. If you don't know the cause, a doctor visit can help identify it. If baby also has fever, fussiness, poor feeding, or other symptoms then see your pediatrician.
Yes. If you're uncertain as to the cause of the rash - and especially if your baby has any other symptoms such as fever, irritability, etc. - the only way to know for sure what may be the cause of the rash is to have it seen by your baby's doc. If your baby seems well otherwise, it probably isn't emergent, but calling your doc to discuss how soon to get seen is important.