Left testicle has shrunk to olive size...should i be worried?

See urologist ...: If indeed your testicular size have diminished for no reason, go to see urologist for detailed general / specific medical history, physical exam, and possible tests such testosterone / LH, etc. So doing will at least give you an assurance. And note: Online without a face-to-face interaction and a direct look and palpation on your testes can not generate the needed specific advice. Best wish ...