If allergic to the olive trees, are you allergic to olives themselves?

No. Allergy to the pollen from olive trees does not cause an allergy to the fruit (the olive). However, you can still have an allergy to both, but by separate and different ige antibodies. Ige is the type of antibody your immune system makes to things which you are allergic and it is very specific for each allergy.