I am suffering with loose stools. not sure if it's an allergy or related to my gall stone.

Loose stools: Most likely due to gallbladder disease. If your gallbladder is not working due to stones, whenever you have a fatty meal you will have trouble digesting the fat and will have loose stools.

Prob no allergy: Though milk intolerance Is possible, meaning you just can't digest it. Food allergy in your situation would be: hives, itching, swelling, vomiting, dizziness, wheezing, sneezing, and, diarrhea later. But not just loose stools. Seek gastroenterologist.