Can your knuckle actually be broken if there is no swelling?

Yes: Jammed fingers can result in simple sprains or complex fractures. Unfortunately the appearance of the finger doesn't always correspond to the severity of the injury. Fractures can be present with little or no swelling. But there is usually tenderness and often pain with motion. If you are at all concerned, see your doctor. A simple exam and set of x-rays is all that is needed for diagnosis.

Maybe: Usually swelling accompanies a fracture. Although the knuckle is a joint and is associated with dislocations which have swelling as well. If there was no history of trauma to the area it most likely isn't broke. Have an x-ray to ensure no broken bones in the hand.

Rarely: That would be unusual, but a bone can be broken with minimum to no swelling.