Should I be concerned if one of my testicles is much larger than the other?

Yes. It may be nothing but you should have it checked out. Some men are born with large discrepancies in size. One testicle may have lost its blood supply at birth and never developed. If it's newly doubled in size, see a physician soon.