My left testicle hurts off to the side?

Fever or no fever: No fever: could be intermittent testis torsion, torsion of appendix testis (onset for either would be sudden), spermatocele (you might be able to feel a small spherical structure above the testis) or varicocele (feels like a "bag of worms when standing and goes away when you are lying down) with fever: epididymitis or epididymo-orchitis (pain onset not sudden).