My doctor said the birth control pill has no effect on future fertility, but I've heard many stories of women having difficulty getting pregnant after stopping. Who to trust?

Majority. If women still conceive after stopping the use of birth control pills or shots or implants.
Birth control pill. Birth control pills suppress ovulation and a number of people stop menses because of this. Once stopping the pill your normal hormones will resume and there should be no problems. If there were problems before the pill then one will be back to their pre-pill status.
Short term effects. There is an effect on fertility of use of ocps, but in most women it is short-term and resolves with time. Best to delay conception for 203 months after stopping ocps. If cycles remain irregular, check with your fertility doctor.
The neighbor effect. You study cars for a year & reports on the data from tens of thousands of reviews say brand X is the best. You meet your neighbor & talk about cars but she says her friend had real problems with brand X. Who do you "trust". Your neighbors 'friend' or tens of thousands of reviews. Your docs spend their lives sorting through such data. The friend may have had the problem with or without BC.
Age is the problem. The birth control pill is meant to stop you conceiving but fertiity returns within weeks of stopping it. I see women in their 30's and 40's who started the pill as teens for painful periods, irregular cycles or birth control and are stopping when they try to conceive. The problem is that it's now harder to get pregnant because the woman is older. The pill doesn't hurt or help egg numbers/ quality.