Mother is 87, suffering from dementia after an infection put her in the hospital in 2013. Does it make sense to continue mammograms in her state?

Probably not. There comes a point where at a certain age, and acquisition of a chronic dementia, one should focus upon palliative comfort support, and start to come to terms with our fond past memories and cherish the prior years. It makes no sense to perform monitor testing for potential breast cancer at this point.
No. If a person is not expected 10 years, there is no need to continue with mammograms. You may consult this site for more info.