Is it harder to gain back muscle after atrophy?

Depends: Muscles atrophy for different reasons. If the atrophy is from say being in a cast for 3 months, the muscles should come back fine with exercise and just routine daily activities like walking if it is a leg muscle. If the muscle has atrophied because of a nerve injury or an injury to the brain or spinal cord, the recovery of this muscle is much more sketchy; it may not recover.