I have mild asthma and am wondering if exercising without an inhaler will make me stronger?

Careful... If you have asthmatic symptoms when you exercise, then you should not exercise without using a bronchodilator inhaler because this would risk an acute exacerbation, which can be life-threatening. See your doctor if you are unsure if you have exercise-induced symptoms.
No-myth. Would you walk on a broken leg ? If you have asthma, you have a defect in your "lung relaxer" mechanism. Once an episode is triggered, it could send you into a significant attack. You don't "get strong" by avoiding treatment, but you do increase your risk of complications.