I have sludge in my gallbladder and have had recent pain there. is there a downside to having it removed?

Unknown: Have a hida scan to see if your gallbladder is functioning properly or not. Removing a well functioning gallbladder may cause a lot of abdominal gas, food intolerances, more diarrhea & abdominal cramping, etc.

Always some risk: With any invasive procedure. Risks of removing a gb and stones/sludge are infection, bleeding, injury of nearby structures (liver, intestine, bile ducts). The risks without surgery would be worsening symptoms and risk of passing a small stone such as developing pancreatitis. It sounds like, for you, the risks without surgery are worse than risks with surgery. I'd likely advise you to have it too.