Can mumps cause a bad taste in your mouth?

Yes. Mumps infects the salivary gland in the cheeks, the parotid gland. Salivary infections can cause a bad taste . One general treatment is to suck on lemon drops to increase secretion form the salivary gland to promote clearing of the infection.
Salivary glands. A symptom of mumps is infection and swelling of the main salivary glands(parotids). It would be reasonable to assume this could alter taste.