What can I do to reduce my oily skin?

Skin care routine. It sounds as if you need a good skin care routine to diminsh sebaceous gland activity and acne disease as well as exfoliate. Too much benzoyl peroxide is not good either and may dry out the skin. Use of a daily facial cleanser such as cetaphil followed by a non-comedogenic moisturizer such as eucerin daily facial moisturizer with spf and periodic topical retinoids and exfoliation may help.
Glycolic acid. Mattifying serums, products that contain aha (alpha-hydroxy acid), glycolic acid, retinol, tretinoin (retin-a) help to control oiliness. Also important not to overscrub or use too much toner or astringent with alcohol in attempt to remove oil--overdrying the skin triggers the sebaceous glands to produce more oil to compensate. Instead use cleansers containing glycolic acid or salicylic acid.