If my baby smells like smoke after being held by a smoker is it harmful even if he is never near so?

Maybe, maybe not. Every person, place, and animal has a scent. The question is whether or not a larger number of harmful particles or molecules gets into a baby if held by a person with a smoke-scent, compared to a person with a roofing-tar-scent, a fried-chicken-scent, etc... Many scents are picked up at work; others are self-imposed, such as perfumes. Perhaps the harm of merely smelling smoky is really small.
Yes. It is not only the actual smoke, but the numerous other chemicals on the skin that can be harmful. In fact, several studies have shown higher incidence in respiratry infections, asthma, and ear infections in children of smokers - even if they never smoke around the child!
2n. Hand smoke. Let me see, well, it's like exposing your baby to a , you know what, you get the message.
I will rec: Thorough hand washing prior to handling a baby. Also consider encouraging relatives/friends who smoke to quit if they want to be around your child. Use of an overcoat/"smoking jacket" which can be removed may be useful for those who must smoke. 1800quitnow is a great resource.