Perhaps not worried
If this is an accurate DX for you brother he has a core difficulty with self regualtion, based on his early relationship hx.
This can often cause distress in adult relationships. Clearly working with a counselor skilled in working with such patients can be helpful. As his sister or brother you can improve your situation with him by being consistent, available and have good boundries.