Is it true that the bigger the visual field defect, the worser the brain has been damaged?

Depends. If the area of the brain damaged is the occipital lobe or the optic radiations, then visual field defects can occur. Generally damage at its maximum occurs immediately and over time there can be a lessening if the areas recover. Of course, the larger the initial damage, the more changes will be detected. Your neurologist can help sort this out.