My ears are always itchy deep inside. My ears are always itchy deep inside. They are very clean and have little to no wax, so that is not the issue. Also, I do not generally suffer from allergies. What could cause this?

Dry skin. You should perhaps have a dermatologist take a look. Sometimes psoriasis can be the problem. Topical corticosteroids can bring you relief.
Bacteria. The external ear canal is a moist place. Normal skin bacteria is located here and can grow to annoying levels, hence the itch. This can be compounded by swimming. I would suggest the otc agent "swimmer's ear" to help clean and dry out the war canal. This should decrease or remove the problem.
Itching . Itching of the inner aspect of the ears is a very uncomfortable condition affecting many paitents. As this is an area of the body not accessible for routine hygiene care, the itching can become very frustrating. Often the reason for the itching is earwax trapped in the ears. Otherwise, a foreign body such as the cotton tip of a q tip can cause similar symptoms. Infections of the outer ear (otitis externa) could be another reason, but this is more of an acute situation. In other patients a condition similar to eczema can also lead to flaky skin or chronic itching. Your best bet is to see a specialist to differentiate which one of these conditions is affecting your individual case, and what is the best course of action.