Is pet dander worse than smoking in the home?

No. While it's hard to directly compare these two, and for someone with a cat allergy in particular cat allergen in the home will be miserable for them, smoking is much harder on your health overall and over time. If you can only choose one, stop smoking.
Worse for what? Smoking in the home exposes not only you, but to everybody else living there to the risks of increased respiratory infection, asthma but also heart and lung disease, including cancer. Pet dander increases allergy and maybe sinus infections, but not the more serious problems. Pet dander is clearly the lesser of these two evils.