I played rugby a few weeks before my fractured ankle had healed, i believe it is fine now with no obvious signs of damage. could there be damage done?

Probably ok: If you damaged or re-fractured a healing fracture, you would feel pain. Bones heal at different times, so the 4-6 weeks that are typically recommended are not 100% right for everyone. That said, it's a dumb thing to do to play rugby when your doc says to give it more healing ime. A secondary fracture can cause big problems. You took a big risk for short term pleasure. Be smart!