Are there any birth defects that are always associated with retardation?

Genetic anomalies: Chromosomal abnormalities are major known causes for mental retardation. Down syndrome and fragile x syndrome are well known examples.

Actual birth defect: The term "birth defect" is usually applied to unusual physical features. There are 2 causes. 1) a problem not due to physical forces outside of the fetus, termed malformation. 2) a problem due to physical forces which have acted upon a previously normal fetus, called deformations. The squashed-appearing face of an infant with prolonged lack of adequate amniotic fluid is an example of 2).