What would you say are more permanent side effects of too much caffeine?

Varies. Most side effects are temporary if one stops, these are usually insomnia, jitteriness, palpitations, nervousness, hyperactivity and diarrhea. But oftentimes if one has been a heavy caffeine user then one would probably get significant withdrawal symptoms like irritability, and headaches as well as fatigue--usually this last about 1-2 weeks then fades.