Is four years old to young for baby teeth to begin falling out?

Not unusual at all. Children will normally begin to lose their baby teeth between the ages of 4-7. The lower two front teeth should be the first to go. As long as the order of the tooth loss and adult replacement proceeds normally, the actual age of loss shouldn't matter. Enjoy!
Very unusual. In a normal eruption sequence lower front teeth are the first to exchange. For girls this usually happens about age 6, for blouse about age 7. Please call your dentist for an examination to determine if your child has an anomalously early but normal exchange sequencing, or if there is a development/eruption sequence. Referral to pedodontist or orthodontist may be in order.