How can you keep a cavity from getting worse than it is?

Difficult. A cavity forms when the hard outer shell of the tooth is worn off or eroded off by bacteria. Then bacteria can enter the dentum, the softer inner core of the tooth, which is a terrific culture medium. The bacteria will then invade unless stopped. Anti infectives can help limit the spread but eventually you will need a dentist to remove the infected portion and cover it up.
Cavity. The only way to keep a cavity from getting worse is to have it filled. You need to have all of the decay drilled out, and then a filling placed.