Is it safe to drink water out of the tap while I am pregnant?

Yes. I don't see any reason why one should avoid properly treated municipal water when pregnant.
Yes. Tap water, in most places in the USA, are as safe or safer than bottled water. However if you have any doubts about the quality of the tap water it should be boiled before drinking.