Mild polypoid mucosal thickening of the anterior right maxillary sinus wall, 0.8 cm found on head MRI, asymptomatic. Should I worry (unilateral polyp?

Agree. Common finding and if not cause of symptoms; tuck it away. All the best.
Not worth worrying. Mild changes on X-rays usually means doubtful change. This implies that you Do not need to do anything until or unless you develop some symptoms related to your affected body part(Sinus in your case). Simply ignore it as a 'nothing of concern'right now.