Is it safe to start exercising for the first time as an older adult?

Two caveats. Yes, but I offer two caveats. (1) have a good medical evaluation to make sure there's no existing or suspected medical issue that would preclude your presently embarking on an exercise regimen. (2) start slowly and build up duration and intensity based on how your body feels.
Yes. But check in with your doc to make sure there aren't problems you might need to address first. Start low and go slow and steady. "strong women stay young" is one of the best books out there for how exercise boosts lives, health, and energy of even octogenarians who've never exercised before and how to get moving and keep going. If you are not a woman...It still works!