I had scarlet fever as child. Now I'm 32, and I have trouble conceiving. Could that be the cause?

Unlikely. Scarlet fever is really no more than a strep throat infection that happens to also produce a rash. It is quite common and is not a known cause of infertility.
No. Scary name but minor illness. Back in the day when we had no antibiotics to treat it Scarlet fever could rage on for weeks of high fever & sometimes transition into rheumatic fever. Even then it had no direct effect on reproductive capabilities. Since the introduction of penicillin, the term has been relegated to the background as just a strep throat with a rash. It has no effect on fertility.