It depends on the mode of administration of oxygen
- whether you're using nasal cannula
or face mask.
With nasal cannula you can, but it may not be feasible with the face mask. It also depends on how long you can tolerate being off supplemental oxygen (depends on how long you shower!) & what the underlying cause/disease process is. Oxygen should be handled carefully because of the fire hazard.