Should they go to the hospital to visit their uncle with dementia if the visit just freaks out the uncle?

It depends. He may be delirious because since he is dementia, he is not familiar with the new environmental setting at the hospital. Make sure he sees and hears well with/without assistive devices, keep at least one or two family members to stay with him most of time instead of many ones, and ask nurses about current medications to compare with home medications.If any of medications are different, just ask.
Hard question. Who are they? Why do you want them to go if he doesn't even recognize them. Who is it benefiting? What do you want to accomplish by the visit? Do others visit without problem?