How can anxiety make you feel physically when you're not having an attack?

Anxiety. Primarily worn down, lethargic, and without energy. Anxiety attacks take a huge amount of energy out of the body, and therefore exhaust the sufferer.
Physically uncomfort. Physical symptoms of anxiety include: a spherical shaped pain in the throat; tightening on the neck muscles in the back; a pressure in the chest making it uncomfortable to breathe; a heavy tight ball sensation in your stomach; an uncomfortable buzz/energy under the skin and muscles that makes you unable to sit still; a tightening of the jaws; heat in the face, cheeks and back of eyeballs etc.
Anxiety can have. hangover symptoms--muscle tightness, headaches, nausea, sleeplessness, etc. There are also different versions of anxiety, from mild and global to specific and intense. You could be anywhere on the spectrum on any of the categories. Go to a psychologist for a look-see. Here's a beginning reference:
Tension. Anxiety is often characterized by tension and readiness for some anticipated event to occur. It can range from restlessness to visible tremors, elevated heart rate and blood pressure etc.
See below. In between anxiety attacks or panic attacks a patient may still experience physical symptoms, even though they don't feel anxious or nervous. The following symptoms are often caused by underlying anxiety: tension headaches, insomnia, abdominal pain with diarrhea or constipation, stomach pain, heart palpitations, muscle tension, fatigue, difficulty concentrating, dizziness and excessive sweating.
Physical anxiety. Anxiety can cause shall breathing, racing heart, numbness in extremities, sweating, dizziness, nausea, fainting, intestinal pain, bloating, dryness of mouth, itchiness ...The effects are manifold and may nee a medical evaluation to make an accurate diagnosis.
Anxiety. Anxiety or worry is a psychological & physiological state characterized by somatic, cognitive, emotional & behavioral features. It signifies fear/concern/unpleasant sensations/ dread. It causes restlessness, fatigue, muscle tension & concentration difficulties. It is considered a normal reaction to a stressor, unless it interferes with daily functioning or causes significant distress, as in panic.
Depends. On the cause of your anxiety disorder, and how frequently you have had panic attacks as well as what medicine if any you take for it. Generally speaking you may feel a bit tired or exhausted immediately after the attack and remain apprehensive.
Generalized anxiety. Anxiety that is chronic or generalized can be manifested in weight loss, cortisol elevation and higher blood pressure and a general feeling of uneasiness. It is generally not a healthy state to be in. It's very important to get treatment from a trained psychologist.