Does anyone know a good way to get rid of keloids caused by piercings?

Several. There are a number of treatments for keloids. We have had good luck with surgical removal of the entire keloid (for example, on the earlobe) and then a series of injections of steroid directly into the scar from the excision beginning 3 weeks postop and continuing every three weeks for a total of 3-4 injections. Other treatments include pressure and low dose radiation.
Keloid treatments... Traditionally include surgical removal as well as injection with corticosteroids. However, other treatments are available. Depending on the size of the keloid and location, surgical removal with close follow-up may be a great approach. Consider asking your dermatologist what approach is best and what can be done to prevent regrowth.