Is it normal for a baby to feel warm after it wakes up from sleeping?

Only if overwrapped: It is not normal for your body temp to go up during sleep. However, if keeping the room too warm, too many blankets or too warm of jammies, then the core temperature could rise. I would discourage giving medication like Motrin in this instance. You do not need to treat fever - it is not a disease, it is a symptom. In this case, just dress down the baby and let temperature come down on its own.