A little bit harmful
Every person, place, and animal has a scent.
The question is whether or not a larger number of harmful particles or molecules gets into a baby if held by a person with a smoke-scent, compared to a person with a roofing-tar-scent, a fried-chicken-scent, etc... Many scents are picked up at work; others are self-imposed, such as perfumes. The harm of merely smelling smoky is uncertain but not big.