Lots of people use the term neurotic casually, what does it mean medically?

Conflict. In psychoanalysis, neurosis is a functional mental disorder that has no discernible underlying anatomic cause. The symptoms -- which can be all kinds of things, including anxiety, phobias, and compulsions, etc -- arise from an intrapsychic conflict between unconscious fantasies and the defenses they arouse. The person simultaneously expresses, and fears to fully express, the impulses.