Toes, fingers numb and the joints seem stiff. At what point should I worry about frostbite?

Now. If you have numbness for more than a few minutes after exposure to cold, you likely have frostbite. If the skin otherwise still appears normal, there may not be much to do except wait. Regardless, you should be seen by your doctor because surgery may be required if you now have signs of carpal tunnel syndrome or other nerve injury.
Look for skin change. Frostbite occurs when the water within the tissue freezes and crystallizes. Injury can occur to the involved area if the exposure is prolonged and the temperature is cold. If you see skin discoloration, or a purple hue, frostbite may be setting in. Its time to limit the exposure and to begin gentle warming. Numbness can be an early sign of frostbite, so make sure to get out of the cold.