Would there be a reason to remove your thyroid gland without you having cancer or a goiter?

Yes: Thyroid cancers especially in adults must be resected. Younger patients do better even if tumor has spread to nodes. Goiters getting large can obstruct the breathing passages and may need surgery, but a small solitary adenoma of the thyroid that is benign but cold in terms of isotope uptake should be removed because of the high conversion rate to malignancy over time.