Should I be concerned about penile cancer or is it too too rare?

Concern Yes. In usa penile cancers are relatively uncommon ( if circumcised even less ) compared to developing world. Any ulcer or lesion on penis must be seen by a physician for presence or absence of penile cancer.
Rare in US. Fewer than 1% of the cancers in men are carcinoma of the penis.
Rare. Penile cancer is rare. You should see a health care provider if you have an abnormal growth or a sore that doesn't heal.