How do you know when enough is enough when trying to get pregnant?

Mostly on you. Some may say that when every available modality has been tried and is unsuccessful, then it is time. It may depend on your financial resources for infertility services. It is a personal decision based on what you are willing to tolerate as far as testing, procedures, costs, and results. Your personal beliefs may play a role. Consultation with your reproductive endocrinologist is also paramount.
Individual specific. Assuming all options are available including ivf, the decision is specific to the social, mental and medical circumstances. Male: azospermia and repeat failure of surgical sperm retrieval. Female (using her own eggs): FSH >30miu/ml, repeat >3 ivf failures, age>46y. Otherwise no absolute reason to stop trying.