What you need to be concerned about is cyptosporidiosis, not cryptococcus.
Both are present in the soil. Cryptococcus affects immunosuppresed individuals. Cryptosporidiosis can affect healthy people but, with infrequent exceptions, it is a mild flu
like illness. Unless you have underlying illness, there is no need to be scared.