Can you be allergic to all molds?

Possible. But highly unlikely. There are thousands of species of fungi. It is highly unlikely that you would be allergic to all of them.