My three year old has a fever of 101 and diarrhea. Should I be worried?

Pediatrician. Yes, please see your pediatrician or primary care specialist.
And what else??? A 3yo is likely to have 3 diarrhea illnesses /yr due to simple viruses & most will be associated with mild fever. If the kid is playful most of the time & taking fluids well enough to pee every 6 hours you may be able to handle it at home. If the kid has bloody material in the stools, acts tired & whiny most of the time or begins to refuse fluids, the kid should be examined.
Yes. Please consult your pediatrician and head to the er.