Is it true that the bigger the visual field defect, the worse the brain is actually damaged?

Yes and no: A visual field defect doesn't have to occur from brain damage. For instance a detachment of the back of the eye(retina) can cause a visual field defect. However, if the cortex of the back part of the brain is damager(occipital lobe of brain) thden the more damage, the worse the eye symptoms.

Depends: Visual field defects can come from brain damage but most commonly from damage to the retina and optic nerve. If the visual nerves in the brain are disturbed by stroke, or vascular changes, tumors or some congenital anomalies, there can be field defects. But the defect depends upon the size of the optic nerves impacted. So a large lesion might have minimal visual disturbance in some cases.