My testicle sack has been growing lately - should I be worried?

No. I wouldn't worry too much. The scrotum can get more relaxed as men age and has little bearing on health. If the testicles themselves are changing, you need a medical evaluation.