Wear cotton gloves
...If needed. True latex allergy is in reaction to airborne latex rubber proteins.
Vulcanized rubber, such as that in bike handles, tennis shoes, rubber bands, etc, does not give off latex proteins. Many people allergic to latex are afraid to be around this type of rubber, but it really doesn't cause any trouble.