I get tiny brown crust on my nipple in the middle. It can be easily remove but keeps coming back. It's soft. Also oily/white buildup not pregnant.

Breast discharge. This discharge could be of no consequence especially if you have recently been pregnant or recently stopped breastfeeding. If not, then you probably deserve some blood work to evaluate your pituitary and your thyroid as well as a possible mammogram.
Check! Your prolactin level. Most likely cause. A low thyroid will also cause prolactin levels to rise! so you also need a TSH !