What exactly is it in the pollen that gives people allergies?

Proteins. Pollens contain proteins that trigger immune reactions in people with a genetic predisposition to develop allergies.