I have a swollen lump behind my right ear. It's not a spot or pimple but it's hard and the size of a pea at the moment. Should I be worried?

Swollen lymph node. Likely it is a swollen lymph node. No you don't have to worry now. Swollen lymph node can happen as a reaction to infection, inflammation, trauma etc. Do you have any earaches, any trauma to the ear etc? It should go away in about 1-2 weeks. If it gets bigger and persists longer than 3-4 weeks- then you go to see your doctor.If you have ear infection or has ear discharge- you should see your doc.
Lymph node. This is most likely a normal lymph node. There are dozens located at different sites around the head & neck & noticed because they are not buried deep in tissues or muscles like they are in the rest of the body. When stimulated by bug bites, skin irritation, vaccines or various infections they will enlarge for several weeks before returning to their resting size.