Is pain in breast something to be worried about?

It depends. Breast pain could be related to the hormonal changes associated with your menstrual cycle or pregnancy. It could also be mastitis which is usually associated with redness or fever. It could also be related with growths within the breast. It is worth getting evaluated by primary care for clarification.