Is IVF ever done without fertility drugs?  

Yes. It is, but much less often. History's first ivf pregnancy was in an unstimulated cycle! there are also a variety of 'minimal stimulation' protocols that involve fertility drugs, but shorter or lower dose regimens.
Yes but ... Yes, but despite early success natural cycle ivf (no fertility drugs) or 'minimal stimulation ivf' with Clomid (clomiphene) or low-dose fertility shots has much lower chances of success compared with 'conventional ivf' with higher doses of injectable medications. Costs are lower, the risk of multiple pregnancy is lower, but results are worse.