Welfare of the pt.
Usually this is a determination made by the doctor rather than by the conservator, but they may have the right to make that call also, depending on the law in your area.
Generally this is done when it is deemed not in the patient's best interest, e.g. If the patient gets agitated or endangers themselves or others when that visitor has been there. Hope it gets better & you can see your visitor.