I can relate. I also used to be very hesitant about mentioning bitcoin because of some bad experiences. But as you said, it feels very lonely. Over time, I realized that the best solution is possibly to just be honestly interested in why people think what they think by asking questions.

For example, you could have asked why he thinks it's a scam.

That also helps you since you not only challenge bitcoin critics (and they may realize that they don't know much about it) but also yourself since you have to know your stuff to teach other people. Teaching other people is the best way to master something.