Are secondary sex characteristics linked to fertility in women?

In a way. Normal fertility requires sexual maturity. Secondary sex characteristics (breasts, hips, pubic and axillary hair ) develop along a timeline to menstruation and sexual maturity. Assuming secondary characteristics have matured by certain age, they are not linked to fertility problems. Absent development can suggest problems, features like breast size or hair growth are largely irrelevant.
Yes. In polycytic ovary syndrome, and other hormonal abnormalities, male type hair type growth or other changes of secondary sex characteristics suggestive of excess androgens, such as testosterone, are definitely linked to infertility.