How can I reduce my jaw muscle size?

Surgery/splint/botox. If your muscle enlargement is due to clench/grinding - a nightguard may help. Botox can also work in the same situation. Occasionally, the muscles that are enlarged can be sculpted surgically by an oral surgeon. See a board certified oral surgeon for options.
It is difficult. As with any muscle, they tend to become more plump when we use them and tend to atrophy (diminish) when we don't use them. We only use jaw muscles for certain things (chewing), so it's hard to 'not use them'. If you grind (bruxism) or clench your teeth at night, the muscles tend to become more prominent due to excessive use. See a dentist about a diagnosis and treatment.