How hard can a baby be shook before it causes problems?

Do not do it. Shaking a baby is never a good idea, even when it is done in gentle playing!
Do not shake a baby! No amount of shaking is guaranteed to be safe. An infant's neck muscles are not yet strong and thus the head is susceptible to movement. The forces generated can have devastating and irreversible consequences, including retinal hemorrhages, blindness, bleeding and swelling in the brain, physical and mental retardation, hearing loss, paralysis, and even death. Do not shake a baby for any reason.