My baby was born 5 weeks early. Should I expect for his milestones to be all 5 weeks late?

It can happen. Babies born 5 weeks early can range from a 5 week delay down to no delay at all. Some of it is influenced by how they did in the nursery and whether they needed nicu care. His doctor will monitor his development at the well child visits.
For a while. We generally adjust our expectations on premi's on a week for week basis for the first 1-2 yrs. Especially if they were very premi (under 30 wks).Many actually show no delay after the first 6 mo, and once they catch up we expect them to stay on track.