Will the delayed teeth underneath baby teeth push the baby teeth out or will it need to be pulled?

Pulled: If the permanent tooth is delayed like you said then the baby tooth would not fall off by its owen . The baby tooth should be pulled to avoid the need for braces afterward.

Usually: Usually the baby teeth will exfoliate or fall out once the adult teeth are ready to enter the dental arch. There are occasions when the baby teeth do not fall out and need intervention by a dentist. There are also cases of not having a permanent tooth under a baby tooth. This is why it is extremely important to have dental x-rays.

Delayed Eruption: Various factors play a role in when baby teeth come out. Sometimes, a baby tooth does need to be removed in order for the adult tooth to be able to come in. See your dentist.

Timing: It's a timing issue and a developmental issue. Teeth are supposed to exchange at specific times and in specific order if the underlying permanent teeth are to erupt into proper occlusion. Aberration in order, timing, or positioning can cause malocclusion. See your dentist. Perhaps an orthodontic evaluation is in order. Do not wait until all baby teeth are lost to see orthodontist.