Could my diet be contributing to my baby's colic?

Yes. There are dozens of research papers published since the 80's and 90's, that demonstrate in some women that their diet may affect the baby adversely with colicky behavior. Lactose problems are very common, but a variety of agents have been implicated. Elimination diets are the best way to determine if there is a problem.
No. Some people believe milk-drinking by a breastfeeding mom can cause baby colic. It is uncertain if this belief is true or not. Perhaps, in a few mothers, some cow-milk proteins get into their breastmilk . . . And somehow cause cow-milk allergy symptoms or colic in their babies. So, it is fine for a mom to stop dairy intake for 2-3 weeks just to see if her baby becomes happier and less colicky.
Yes. Certain food have been shown to make babies gassy or fussy. If you think your diet is affecting your child, some common culprits include caffeine, high fiber foods (broccoli, cabbage, onions, etc..) and foods high in protein such as dairy, eggs, nuts, soy, wheat and seafood. It can take several days of removing these foods from your diet to see benefits.