I don't have the sickle cell trait but the father of my baby does, so what's the worst that could happen to my baby?

Trait ++. If we assume that neither you of babies dad has any other gene problems, baby has 50% chance of getting trait. Baby also has the 4% chance every baby has for having some unexpected issue at delivery, including prematurity, illness, need for ICU care, birth injury, etc.