How long should I expect swelling after removing cast off hand (fracture, cast lasted 4 weeks)?

Long Time. It is normal for you to have sorness, swelling and stiffness in your hand for many months. Occupational therapy can help get this worked out faster.
Everyone's different. Swelling can persist even after cast removal. Your time course was relatively short and continued swelling could indicate that you have not completely healed. It could also be as simple as your body's natural reaction to trauma. If you have pain as well or if you are concerned, speak to the doctor treating your fracture.