Should I be worried that my baby is hairy?

Yes. If the hair does not go away after several months, or is limited to a tuft of hair on the lower back, bring this to the attention of your physician. The first may just be familial, or could be related to a number of problems. The second could indicate a problem with the spine.
No. No, not unless everyone in the family has very little body hair, and nobody had any body hair as a baby or as a child. So, if the baby is the only member of the family who seems hairy, then there may be a problem . . . And the baby should be seen by the doctor. If the rest of the family is "hairy", then the baby's "hairiness" is may be hereditary; but let the doctor look at the hair to be sure.
No. Babies are born with a fine covering of hair called lanugo. It is more obvious on some babies than others. It is darker and thicker on certain ethnic groups, such as those of mediterranean descent. This hair usually goes away after a few months.