I have minor testicle pain, no lumps or swelling of testicle noted by me or my dr. he thinks it's referred muscular pain, does that make sense?

Yes, makes sense: Most relatively mild or vague pain involving both testicles, with little tenderness, is not due to actual testicular problems, but is referred pain -- from the prostate or, most often, tension in the pelvic muscles. Probably this is what your doctor is referring to. Google chronic pelvic pain syndrome (there's an excellent Wikipedia article) and see if the symptoms match yours.