Extracted tooth but small piece is still remaining, is it harmful?

Probably Not: Depends on the size of the piece. If it is small, it may not be harmful.

Generally Not: Small parts of teeth break off often, but they rarely cause problems. In fact, going after the small part and trying to remove it often causes much more harm than just leaving it in place. So, don't worry about it. Keep a close watch on the piece with regular x-rays (radiographs) and have your dentist evaluate it regularly. If it becomes a problem, simply have an oral surgeon remove it later.