If the breast tumor is shrinking with chemotherapy is there a chance not needing surgery?

No. Cancer cells are very small. Tumor can shrink dramatically as a result of chemotherapy. As a matter of fact, most do! and some patients have their tumors shrink completely - and when surgery is performed there is absolutely no evidence of any residual cancer. But you can't be sure unless you operate. For who would want to risk leaving even one cancer cell behind? Not me.
You need surgery. The standard of care is to remove the area where the tumor was located if the chemo causes the tumor to "disappear". There are many surgical options. Speak with your surgeon and plastic surgeon. It is great that your tumor is responding to the treatment. Keep the course, you're headed in the right direction.
Neoadjuvant chemo. Chemo given after the biopsy and before surgery has been found to help shrink some women's breast cancers.It will often allow the surgeon to take less tissue at the time of surgery.It is not a replacement for surgery.Even if there appears by exam to be nothing left, pathologically there will be a tremendous number of cancer cells left.Have faith in your doctors...If you don't, find ones you can!