How does certain type of fabric affect your skin health?

It depends. Some fabrics can cause irritation of the skin (wool) or even allergic reactions (man-made fabrics). Dyes in fabrics can also cause problems in some people. If you think you are having a problem with something in particular i recommend you see a dermatologist.
Allergy. You can be sensitive to the color or other chemicals used in the fabric.