You're right, you never know what kind of person you might run into. In my opinion, a good strategy is to have a wallet with some bitcoin and hand that over. Minimize the losses!