I don't wear a bra most of the time. does this contribute to boobs sagging?

No bra=sag?: Hard to respond to. Just be physics the excess weight of the breasts with gravitational downward forces could contribute to "saggy" breasts.

Breast Sagging: Breast sagging occurs as ligaments connecting the skin and glandular breast tissue stretch. Some people have stronger ligaments than others. The larger your breasts the more likely to sag as there is more weight stretching the ligaments over time. A supportive bra may help that. Pregnancy can also affect sagging. In general a smaller breasted person would have less sagging. Hope that helps.