I was playing soccer 3 weeks ago and was kicked just above ankle, bruising gone but still swollen/tender, should I be worried?

Probably Not. Contusions in this area often lead to a hematoma (collection of blood) which may be slow to resolve. Provided there is not any pain, the patient may return to activity as tolerated, taking care to protect the area with a pad/shell to prevent re-injury.
RICE. Try the rice method, and progress back to your preferred level of activity slowly and as tolerated. If you fail to improve, see an orthopedist for x ray and evaluation.