My eyes get really itchy and irritated when I take my contacts off. Is this normal?

No. Could relate to contacts lens issues such as allergies, poor fit, overwear, solution intolerance, giant papillary conjunctivitis, etc.
No. You shouldn't have much reaction when you take your contact lenses off at the end of the day. Sometimes, chronic allergy to contact lens use and solutions can irritate the back of the eyelids which can give an itchy and irritated sensation when the lid is back in contact with the cornea. Good idea to be check out for fit and allergy.