Older man wanders. Sometimes I can't control and he gets in an neighbor garden patch. They don't mind. Is it ok?

Safety first. If he just wanders calmly & isn't putting himself or anyone else in danger (getting agitated/combative, wandering into road or traffic, etc) he will probably do better at home in a familiar environment getting care from people who know him well & love him. But, if he is putting himself or others in harm's way, even if its unintentional, he may need more help. A geriatrician can help guide options.