Swollen eyes went away and came back. Could they be due to cat allergy?

Absolutely. Cat dander is a leading cause of eye allergies. If you experience itching red and swollen eyes during and after exposure to a cat, you are likely allergic. Allergy eye drops and oral antihistamines will improve your symptoms relatively simply, but the best treatment is avoiding exposure to cats.