Is it harder to gain the muscle back after losing it?

I don't thinks so: There is something called muscle memory. Muscles that were once highly trained tend to respond to re-training than muscles that have never been exposed to training. It may feel harder because it is hard to focus on where your performance is currently without applying the context of your prior abilities. Meaning, it is frustrating to get back into the routine.