How do doctors determine if it's rotavirus or norovirus?

Doesn't matter. Although virologists can distinguish these, clinically it doesn't matter which a person has. Both are highly contagious, diarrheal organisms that are typically nonlethal but can put vulnerable patients, esp the very young and very old, at risk. Both are treated supportively with hydration, and prevention of spread is key.
By viral culture. But you can make guesses as to which virus it is. Norovirus occurs all year round. Rotavirus is in cooler months ( October -April). Rotavirus very common in children. There is a vaccine for rotavirus but not noravirus. They are both miserable illnesses with vomiting and diarrhea. .