Can you have permanent damage from b-12 deficiency?

Yes. Prolonged B12 deficiency can result in neurologic damage which is at best only partially reversible.
B12 deficiency. Unfortunately, the brain and neurological manifestations of B12 deficiency can be permanent.