I feel like my mother has serious mental problems, but can't point to specific symptoms. Something is just off. What do I do?

Professional eval. Sounds like she would benefit by being evaluated by a qualified professional, such as a psychiatrist, psychologist, counselor or even family doctor or internist. Hopefully you can accompany her to help assure that your concerns are addressed during the visit. It may help if you can explain better why you think she is having serious problems. Hopefully you can persuade her to be evaluated.