Ask around. There are family docs who are great at it and some urologists or endocrinologists who aren't. Find a doctor who has an interest in it and you'll probably do better.
Family docs, too! Please understand my bias as a family physician as I believe that we can safely diagnose & treat men & women suffering from hormone-related conditions, such as diabetes, hypothyroidism, menopause & hypogonadism. In the residency where I teach, it's very common for us to treat men w/low testosterone. I lecture on this topic both state-wide & nationally. But if it's more complicated, go see urologis.