Should I worry that my baby sleeps so much?

No. Given that it is normal for newborns to sleep 16 - 20 hours per day they often seem like they are sleeping all of the time. Their sleep is often divided evenly throughout the day and the night. Many infants need stimulus to stay awake even during feeding.
No. As long as your baby is regularly waking up for feedings sleeping a lot is nothing to worry about. As your baby gets older they will sleep a little less and be wakeful for longer periods of time. Around 4 months babies often begin to consolidate their daytime sleeping into 2-3 naps with longer, more active wakeful periods in between. If your baby doesn't seem to wake even to feed see your dr.