Can you be allergic to condoms but not due to a latex allergy?

Yes. Most codoms have a lubricant which you could be reacting to. Some condoms are latex free, try this type to see if they cause any problem.