My vagina is getting red itchy bumps on the inside. Should I go to the er?

Yes. Better to be safe than sorry since this is a mucosal tissue and it could be an infection, allergic reaction, etc...