You need to talk with father and explain if he leaves evidence that he is using pot, it is a bad influence on younger children
If he does not think this is reasonable, then he is acting unreasonable. He should not expect to to visit and exhibiting irresponsible behavior with your children. Better for you to visit alone, if doesn't grasp this. Never compromise with bad behavior!