Phobia of eating other people's germs with my food keeps me from parties and restaurants. Am I crazy or smart?

Cautious vs Phobia. The key to whether an avoidance behavior like yours is a disorder is whether it compromises your life in significant ways or prevents normal life functioning (in work, relationships, school, etc). If the phobia of eating other people's germs has affected friendships and your life in general, it may be a good idea to seek an evaluation.
In the middle. It's smart to want to stay healthy and avoid germs. However, it crosses the line when your anxiety or phobia of germs interferes with your work, relationships, family or when you start avoiding things that you would otherwise enjoy. Talk to a doctor and/or therapist and get evaluated.