What happens if I don't keep the cast on my broken hand?

Not a good idea! Taking off the cast before the bone heals can lead to non-union (fracture doesn't heal) or mal-union (heals in the wrong position). Either of these can lead to permanent arm disfunction.
No. Do not remove except when md says. You'd lose the benefit of having it, namely stabilizing the bone in proper alignment for it to heal that way.