Is there an over-the-counter treatment for Bartholin's abscess or cyst?

No. The bartholin's duct cyst--abcess sometiimes can be opened and drained, but many times, to prevent it from returning the gland is surgically removed completely under anesthesia.
No. The treatment can depend on one or more of these factors: the size of the cyst, how painful it is, if it is infected, and the patient's age. In some cases, a small cyst can simply be observed over time to see if it grows. In other cases, the doctor can perform a minor procedure in the office, in which a small tube is inserted into the cyst]and inflated to keep it in place.