How would you know if your arm is broken?

Pain, function loss. An xray will confirm a fracture, but if you have had a significant trauma and you have pain specific to a certain area, with a visible and/or palpable deformity, tenderness to palpation and an inability to perform regular joint motion, then you very well may have a fracture. Go see an orthopaedist and find out for sure!
Pain. Pain, swelling, in ability to move, deformity can all be a sign of fracture; most fractures can be seen on X-ray, although not all.