Could it be something bad if I have ear swelling?

Unlikely. Ear swelling is not very common... But most likely related to otitis externa - outer ear infection.
Maybe- call your Doc. Have your doctor check it out- 1. Pulling on the ear lobe may trigger pain that could represent an outer ear infection (otitis externa) which may require antibiotic/steriod drops and possibly, antibiotics by mouth. 3.If the ear is tender and bright red, it may also represent an uncommon condition called relapsing polychondritis (treatable). 4.A skin infection is possible (treatable).