There is no one answer to if or how harmful infidelity is, but, ultimately, it almost always is.
I believe there a paradox built in to every life. Homo sapiens are not--by nature--made to be with one sexual partner for life (especially the male species). Thus, the human-made institution of marriage--the most potent a social arrangement that exists-is, to say the least, difficult. Life is a stage.