I'm five weeks pregnant. I had a scan today was told I have a dermoid cyst on my only ovary. How does this affect my pregnancy + should I be worried?

Very Common. Dermoid cysts are very common in reproductive aged women. They are nearly always benign. If they aren't causing you pain, i bet your doctor will recommend waiting until sometime after pregnancy to consider removal. Surgery during pregnancy does occur, but only when absolutely necessary. The cyst will likely not affect the pregnancy in any way.
Dermoid cysts. Are benign and should not harm an intrauterine pregnancy. Dermoids are slow growing (1-2 mm a year) and therefore some advocate non surgical management. Larger lesions are often surgically removed as twisting of the ovary and cyst rupture can occur. Many recommend initial serial follow for lesions under 7 cm to monitor growth, beyond which a resection is advised.