Can physical health suffer from spending too much time indoors?

Yes: You require sunlight for vitamin d metabolism, which helps control calcium levels. You probably don't have fresh air, likely you have decreased exercise, etc. So, while "getting outside" by itself may not be required, it is the total effect of natural light, open air, no air conditioning, no re-breathing old air, etc, that has a beneficial effect as you walk around. Try it!:).