Can one time moderate use of cocaine lead to permanent brain damage, memory problems or cognitive impairment?

Death = permanent. In some, the first use of Cocaine causes death, which would be permanent brain damage or impairment. Not to mention heart attacks, respiratory failure, strokes, or seizures. In the brain alone, Cocaine is a very strong CNS stimulant that stops Dopamine from being recycled -- making it build up and disrupt normal neuronal communication. If you're dead, the long-term brain changes don't matter.
Least ofyour worries. Although all the things you mention could occur with one time use of cocaine, however, these are nothing when one time use can cause death!