Should non-diabetics really be concerned about their blood sugar?

Yes: At some point, all patients with diabetes had normal blood sugars prior to getting diabetes. If you're blood sugars are >99 mg/dl fasting or >139 mg/dl 2 hours following a meal, then such abnormalities may imply that you have pre-diabetes or even diabetes depending on how high your blood sugars might be. Even low blood sugars, < 65 mg/dl, may be a risk factor for diabetes.