Will there do a huge difference when switching from toric lenses to spherical?

Depends. Toric lenses are to correct astigmatism. Spherical do not do this. If you have astigmatism and switch from toric to spherical, you simply will have a less clear image. If the astigmatism is small, it might not matter but if it is larger, you might appreciate the clarity from the toric lense.