The ear swelling is on the inside. I can fit an earphone comfortably in my left ear but only get it halfway with my right. Could this be swimmers ear?

Possibly. If there is no discomfort and this is not a new symptom then it is likely not swimmer's ear. It could be something as simple as an earwax impaction. A physician's evaluation should easily make the diagnosis.
Anatomic defect? Swimmers ear , you get oozing, discharge from your ear canal. Perhaps, your right ear anatomy is different, and therefore it doesn't fit well.