Love goes where sent. That may seem odd but think about it. Love is a choice we make based on the experiences we have with each other. That may happen sooner or much later in a relationship. A great many feelings are involved in the choice and come from that relationship. So we choose love. The reverse is equally true, when negative experiences dominate. A few good things left make it hard to let love go.
Love is forgiving. Read the books- four agreements by miguel ruiz - just listen do not agree to words, read the book -when love goes wrong by ann jones. Love does not mean you condone a behavior - there is nothing worth that blocks our inner bliss. So be always happy inside your heart and unroot the words from your heart every day and imagine tossing them all the way into the oceans (oxymetazoline).
Listen to your head. It seems as though you feel like you need to fall out of love with him in order to leave him. I don't think that is a realistic expectation. I can't say whether or not you'll ever really "fall out of love" with him. There are powerful unconscious forces at play. You will need to follow your head on this one and your heart will follow.