They say my eyes changed from grey to brown when I was a baby. Is that really possible?

Definitely. Many children, even those of black or hispanic ancestry, are born with light colored irises. The basic model iris is actually white, though we call it "blue". All the other colors like green, hazel, brown, and black are result from adding different amounts of brown pigment to a white iris. The irises of many children darken with age.