Can swallowing sperm be harmful?

Not likely. Sperm/semen of a healthy male, without transmissible diseases, is not likely to cause any harm.