What is the chance of catching herpes or any std if someone scratches back of your hand by his finger nail during playing basketball? i suspect that person might have touched his mouth during the game?

HSV1 and HSV2 (the ones that cause oral and genital herpes) typically spread through contact with broken mucous membranes (such as lips/gums and genital area). If a contaminated fingernail scratched the back of your hand, it is very unlikely that it will spread there unless there was a large gash on your hand and there was a lot of virus on the dirty fingernail in question.

Practically none. Herpes is not spread that way.