Can I give my 5 week old infant water or should I just give him his milk?

Breast milk. I would advise not giving water at this age. Breast milk and/or formula contains water and nutrients for growth. A small amount of water eg 2 oz in one day is safe to give, but there is no need.
Either. Water is a healthy part of any diet. Formula or breast milk is actually balanced to provide the proper amount of water equivalent, to the baby making extra water unnecessary in most cases. It is a reasonable supplement to breast fed kids when relatives want to share the joy of a nurturing moment, & extra water for a kid with hard stools is often helpful.