Is it actually harmful to have cold hands/feet, or just annoying?

Mostly annoying: It could represent reynaud's phenomonon where blood flow gets reduced in the cold dramatically. If mild, it is annoying. In the most severe cases, especially when exposed to severe cold, the lack of blood flow could actually cause damage to tips of fingers, and toes, but this is with the worst cases and the worst cold. Most of the time, just having cold hands is just that and warm gloves help.

Either: It depends on the severity of 'coldness to your hands or feet'/ in most cases, treatment can be as simple as keeping your hands and feet warm by wearing gloves and warm socks in cold weather. In worst cases, their are rheumatologic conditions like scleroderma or lupus, which can actually cause ulcers at the ends of the fingers or toes. If there is pain associated with cold, have it checked out.

If you have Raynauds: Either the syndrome or the disease, you would have to be careful when going out in the cold as you are at a higher risk of developing something like frostbite or childblains.