Can stress or anxiety lower testosterone in men?

Defitely yes. Testosterone levels can decrease with acute or chronic stress or anxiety. Theory is that the stress or "flight and fright" hormoes, epinephrine, nor-epinephrine and cortisol. This lowers the anabolic (tissue build up) process and inhibits testosterone production. This is reversed after stress and anxiety are relieved.
Some interaction. The interaction of hormones is extremely complicated. The pituitary gland and hypothalamus in the brain, the adrenals, and other organs such as the testes are involved in regulation and various checks and balances in testosteron. Stress can definitely impact the effects of testosterone as cortisol and testosterone are parts of fight ot flight reflex. Chronic stress has many negative hormonal effects.