Should I go to the doctor if I get pimple like lumps under my armpits?

Maybe. These are common. If you are having large or painful lesions, or very frequent problems with this, consider seeing your pcp. If you have small bumps that come and resolve on their own, medical attention may not be necessary.
Lumps. You may get more definite treatment by going to your dermatologist.
Oral Antibiotics. Pimples under the arms can be folliculitis which can be caused by a bacterial infection. Thus, you may need oral antibiotics to control this condition.