I am assuming you became a vegan for philosophical reasons along with health benefits.
This would include, therefore, animal exploitation which the extreme vegans incorporate in their lifestyle. Polo horses are generally well cared for and seem to enjoy their lifestyle. But if you feel that their use is exploitative, you are within your rights. There is no absolute right or wrong for this.