After wearing colour contacts without diopter my vision is blurred when I don't wear them?

Contacts need to fit. It is very important that an eye care professional fits your contact lenses, even if they are only worn for cosmetic purposes and don't have any "power." you can get serious eye issues if the lenses are not fit properly to your eyes. If you haven't been fit or evaluated by an eye care professional, please do so. I agree it could also be dry eye.
Possible dry eye. Contact lenses of any type can result in dry eye and cornea inflamation. Make sure that you are not sleeping in your contact lenses and not overusing them. You also may want to try over the counter artificial tears which may help with your vision as well.
Oxygen! You may be experiencing a problem called hypoxia, where not enough oxygen gets to the cells of the cornea. Patients will often say that their vision is foggy, like after swimming with their eyes open. The lack of oxygen causes swelling of the cornea, which is the surface where your lenses rest on your eye.