How do you know if you have a family history of colon cancer if all the old folks get colonoscopies, get polyps removed, and say nothing about it?

Ask. Go to your older relatives & ask them- not just about colonoscopies, but about their parents/grandparents, who had what & who died from what. You'll get valuable family history on a number of topics. Only about 5% of colon cancers occur in families with a history of colon cancer, so screening is more important than family history for this.
Not all family equal. The family history one is most concerned about is first degree relatives less than 60 with an advanced adenoma (large polyp) or cancer. They same applies for 2 second degree relatives. Even if you grandfather had cancer at age 70 it doesn't change the screening recommendations for yourself.
Great question. If your family won't tell you their history, then there is no way to know your family history. Keep asking and explain the importance.