It's you genes. About 5% of men, myself included, can't grow a beard or chest hair. Many women prefer this, and guys like us usually don't go bald. Joel kinnaman (failed beard, photo), muhammad ali, paul newman and larry bird are all smooth-faced guys. They don't seem unhappy over it. Your genes are your heritage and for you to enjoy. Remember a guy looks good if, and only if, he's physically fit. Cheers.
Stop shaving. When a person stops shaving the face, hair will grow out as fast as it is destined to. Some men grow facial hair quickly, but others do not (this is almost always a genetic trait). If a man has very little facial hair compared to his relatives, he can see his primary care doctor for an evaluation to see if he has a medical problem. Of course, a woman with excessive facial hair must see her doctor.