Could one have mastitis even if you're not pregnant or breastfeeding?

Yes. Yes although it wouldn't be terribly common. If you have redness, heat, swelling, burning - any of the above see a doctor and get checked.