I feel like an idiot for asking this question, but...?

Spermatophobia is the fear of germs. Xenophobia is the fear of strangers or foreigners, Aphenphosmphobia or Haphephobia is the fear of being touched. Irrational fears are very common, you're not an idiot, but you might benefit from help. There are lots of good counselors out there. Good luck.