How worried should I be about cat litter?

Yes. If you are pregnant, you should avoid handling cat litter. If you are not pregnant, just wash your hands well after changing the litter.
Toxoplasmosis! The biggest concern with cat litter is the risk of toxoplasmosis. This is definitely a concern in pregnant women. If you are pregnant, you should not be handling the cat litter. For infants and children, of course you don't want them playing with cat litter. But if you find your child playing with it, wash their hands well.