Should I worry if my eyes are unequal sizes?

No everybody's are. Everyone is asymmetric, that is, diference between the sizes of arms and legs and organs in each side of the body. What you have noticed is a normal finding.
Unequal eyes. Your eyes are likely the same size. One just appears smaller due to droopiness of the lids on one side. This can be due to many different issues so you should see an eye doctor.