How quickly would the brain have to be moving to hit itself against your skull?

You'd be surprised: Experimental studies evaluating the brain's displacement during a head-on injury have tested speeds as low as 2.5 m/sec or 5 mph and shown that the brain will move a few mm (reference: http://www.ncbi.nlm.nih.gov/pubmed/1691964). Modern imaging including MRI demonstrates that even if the brain doesn't move a lot within the skull, diffuse axonal injury from shearing forces can still occur.