What could cause an abscess on the breast if you're not breast feeding?

Many possibilities. Breaks in skin, however minor, can cause infection in that area. Sometimes it may be the result of blocked ducts, but rarely this is a manifestation of infection elsewhere extending through the bloodstream. If an echocardiogram has been ordered, your doctor is thinking systemic disease spreading. Endocarditis (heart valve infection) is where they are going. Follow up assiduously and get better.
Boil. Boils are referred to as abscesses, in the medical community. They can occur from breaks in the skin, which provide a route of infection for bacteria to form an abscess (collection of pus/infection). When this happens, you may need to see a medical provider who may need to prescribe you antibiotics or incise and drain the abscess. Some home remedies may include a warm compress and ibuprofen.