Maybe, maybe not. If the tumor is benign then it might not need to come out. However, if it's in there long enough, it might start to cause problems like blocking the ducts to your mammary glands. Also, just because it's benign today doesn't mean it won't transform at some point. It might be worth having it out, if it's simple and easy to do. See your local surgeon for more ideas.
Let me explian. Intraductal papillomas can be surgically removed if they prove bothersome. A small incision is made along the edge of your areola, then the papilloma and its duct are removed. The resulting scar can be nearly undetectable.