I heard people back in the day that used baking soda to whiten their teeth. Does this work?

Not Really. If you have stain on your teeth the baking soda can help remove extrinsic stains(on the outside of the tooth). I believe however you are referring to "tooth bleaching" which is the process of removing intrinsic stains and baking soda is not very effective for that purpose.
Better Options Today. Baking soda is abrasive and should not be used to "bleach" or clean teeth very often, or at all. Abrasives seem to clean teeth well, but actually go too far, and can cause abrasion of good tooth structure and loss of tooth surface. Stick to non-abrasive dentifrices, and chemical bleaching with carbamide or hydrogen peroxide from your general dentist. It's fast, inexpensive, and does no harm.
Can damage teeth. Have a dentist see what you now have. Then they can guide you in the best way for you to whiten.