How concerned should I be about my baby's fever?

Look at baby first. I recommend parents "look at the child, not just the thermometer". Regardless of temp, if you are concerned call your doctor. If your baby is less than 3 months old and the temp is 100.4 rectal or greater call your doctor. Children with fevers that don't respond to fever medications, last 3-5 days, or have associated symptoms (such as sleepiness, fussiness, poor intake, rash) should also bee seen.
Very concerned. The main cause for fever is infection and it's important to identify the focus. Your baby needs a thorough work up, including a physical exam and if indicated, lab tests. Treatment and work up depend on how sick the baby is and his or her age. Take your baby to see a doctor.
Depends on the age. Any infant under 1 month of age with fever 100.4f (38.0c) should be seen by a doctor as soon as possible. If your infant is older, it is wise to at least contact your doctor's office as fever is typically a symptom of a viral or bacterial infection. As an infants' immunity is much lower than an older childs', fever could be a sign that your baby may need treatment to help him/her fight the illness.
Fever. More important than fever is your interpretation of the baby's condition. For example, he may have a high fever and be a jovial babr, eating, smiling, cooing, etc. On the other hand, a baby may be afebrile and be very ill. If, at any time you are concerned for any reason, pick up the phone and talk to your pediatrician. He will not only have a calming effect, he will tell you if the baby needs to.
Forget fever.... ..How concerned would you be about your baby's appearance if you never took the temperature? Is he cranky, listless, lethargic, pale? If you didn't think he was that sick before you took the temperature, he is still not that sick regardless of what the thermometer says; and if he looked sick before, do not be reassured by normal temperature.