I'm a 20 year old male, I've smoked cigars on and off for around 4 years now, how much damage?

Minimal. Smoke related damage to the lungs is cumulative over time. 4 years of tobacco smoking is quite minimal unless you have a positive family history of lung disease or a genetic predisposition to developing lung disease. Short term effects of smoking include lack of taste, lack of smell, discoloration of the teeth, lack of libido. Longterm effects are greater including cancer, heart attack & stroke.