Maybe, maybe not
If the tumor is benign then it might not need to come out.
However, if it's in there long enough, it might start to cause problems like blocking the ducts to your mammary glands. Also, just because it's benign today doesn't mean it won't transform at some point. It might be worth having it out, if it's simple and easy to do. See your local surgeon for more ideas.