Is it safe to start dieting as my baby is breastfeed but only twice a day as trying to change to formula?

Probably: You didn't mention how many months you are post partum. In general you can "start dieting" around 3 months post partum. But if the baby is less than a year, I'd encourage you to get more calorie deficit for the buck, kill two birds with one stone, continuing your twice a day nursing till the baby is a year old or more. Those calories are coming out of your body, nursing helps you lose.