I have never worn glasses and my vision is pretty good (but not perfect) - am I making my vision worse by not getting glasses?

Not at all. Glasses do not strengthen or weaken your vision; they simply provide your eyes with better focus. You may see and feel better with a pair of glasses (especially when driving), but as long as you are comfortable and your vision is within your state's legal limits for driving, you do not have to wear glasses.
Okay. You don't harm your eyes by wearing or not wearing glasses.