It boils down to comparing risks to benefits.
How impairing and/or distressing are the symptoms? Also, rather than thinking of starting meds as a forever "marriage", think of it as a "trial". If the med is not well tolerated doesn't and provide significant improvement, stop giving it! despite anecdotal horror stories, there isn't any convincing evidence that a brief trial risks any permanent harm.