Milk/dairy has protein
and may also have fat [unless you're drinking non-fat].
Both protein and fat take longer for you to digest. If your body is busy digesting, there's less blood flow to the moving muscles in your arms and legs. Less blood flow leads to cramps.
Ideally you can wait 2 hours after a big meal, 1 hour after a lighter meal before exercising. Congrats on getting out there!