How do I know my toe is broken? When do you need an x-ray?

You don't. Unless there is severe deformity of the toe an x-ray would be needed to see if it was fractured.
Not improving. If the toe is misaligned or not improving, have it evaluated. Letting injuries go can lead to more problems at times. See your podiatrist. Dr l.