Which medication is more effective in treating insomnia, promethazine or diphenhydramine? Which one causes more sedative effect?

It depends. The best way to treat insomnia most of the time is with improved sleep hygiene - avoid late day naps or caffeine, only use your bed for sleeping, and ensure you have a proper diet and exercise. Sedation is a side effect of the medicines you mentioned, patients may feel more sedate from either one depending on many things.