Can mold allergies make it so you can't eat certain foods?

Possibly. Mold allergy is from allergy to the mold spores in the air and not to the mold itself. There are some foods that may have mold or are made with mold. Cheese, mushrooms, vinegar, sour cream, sour milk, buttermilk, beer and wine may all have different types of mold. http://www.livestrong.com/article/542511-safe-foods-to-eat-with-a-mold-allergy/
Mold and diet. Yes, in more serious cases, mold and yeasty products can affect you such as vinegar, fresh baked bread etc. Go on the net and there are lists helping you reduce mold and yeast in foods. See if you feel better by following this for a few weeks.