Do people have a right to physically restrain you if you're not physically violent?

Depends. If you mean ordinary citizens, they do not have such a right (parents excepted). If in some official capacity, they might have the authority (such as tsa officers or prison officials, or police).