This one guy starting flirting with me and got my number and we've hung out a few times but now I do all the initating, is he not interested?

Yes. It seems that something turned him off, maybe something was said during those hung out sessions or did not show a positive reaction to one of his comments.