Why is it that Benadryl (diphenhydramine) gives me a hangover?

Crosses into brain. There is a blood:brain barrier and medications can cross through this barrier to cause people to feel "hungover". People complain of feeling drowsy or paradoxically some people get insomnia. Non-sedating antihistamines (fexofenadine, loratidine, cetirizine) have less of this "crossover" and so cause less problems for people.
Sedating effect. Benadryl (diphenhydramine) and some other older generation antihistamines cross the blood-brain barrier, thereby providing a drowsy "hangover" effect. This is usually limited to 4 to 6 hours, once the drug washes out of your system.