Is there a way to check if we are allergic to milk?

Allergist. To definitively know about an allergy, an allergist will need to perform tests. Prior reactions are a good indicator, but the tests can give a definite answer.
Clinical symptoms. Must be apparent in conjunction with positive testing, either percutaneous or serologically.
Yes. See your pcp or allergists. The will first take your history and perform an examination. If your history suggests a food allergy to milk then they offer you skin testing and / or blood testing for milk allergy. Note all food allergy diagnosis is dependent upon your history. Testing alone does not tell you that you are food allergic.