Why do I have man boobs even though I'm not fat?

Many causes. There are many causes for this. Some of these are serious conditions that need to be evaluated and treated accordingly. These causes include: malnutrition, alcoholism, liver problems, drugs (legal and illegal) side effects (spironolactone, aids meds, marijuana, amphetamines, ), prolactin overproduction, low testosterone, kidney failure, thyroid problems....
Not uncommon. Many patients with gynecomastia are not overweight. It's caused by a hormonal imbalance usually. Many young men have a mild case of gynecomastia, with only puffy nipples. Others have a more serious case. Most require surgery for correction, even mild puffy nipples. See my webinar on youtube - search dr. George pope. Or visit www.Gynecomastia.Org, a great forum.
Drugs/alcohol/none? Multiple reasons. Usually some medications, drugs, alcohol are behind this issue but it may present without any particular cause.