Even if they're not bugging me, should i consider getting my wisdom teeth removed?

Yes: Yes, "considering" is highly advised: it is not that simply removing all and any wisdom teeth is a good idea. If you are one of the small group of people who have healthy, functional wisdom teeth in sufficient, hygienic space, keeping all 32 teeth is good. However, if such is not the case, weighing the benefits vs risks (present ; future) with your doctor will help you decide what is best for you.

Depends: Depends on the situation. Often the wisdom teeth are causing problem jeopardizing the adjacent teeth. Best advice is to see an oral surgeon for exam/recommendations.