Expert opinion please: could taking allergy medication cause memory loss?

Usually not so. But, in the elderly, the anticholinergic effects of anti-histamines can worsen chemical deficiencies, and make cognitive function appear far worse. This is merely a transient effect, not a permanent lingering problem after medication is stopped.
No. It's great to take allergy meds and be rid of those irritating symptoms. All the best.