Yes you should be worried about an infection.
Acute swelling and pain can also occur from bleeding into the knee. If you are on any anti-coagulants you can bleed into your knee even if there was no specific injury or trauma. Either way you should see an orthopedic surgeon asap. Waiting only allows an infection to become established and harder to treat.