Is there a substantial difference between having an addiction to sex or simply having poor moral character?

Probably. Addictions of any kind involve more than just temptation and moral weakness - there appears to be a vulnerability to addiction that is due to how the brain's Dopamine / reward system works. Plenty of folks with poor moral character never develop addiction. And some people with otherwise good moral character fall into addiction.