Will smoking pot have an impact on our ability to conceive?

Yes, it can. Depends on how much and how often you smoke. Particularly in men, cannabis reduces sperm counts, and sometimes increases numbers of abnormal sperm or sperm that does not move correctly. It raises estrogen levels in men, which may reduce fertility. It has fewer effects on women, but smoking marijuana while pregnant is not safe.