You're absolutely right, and the problem with using commercial vendors like Yubikey is that you cannot deterministically recreate your key on another device, should you lose your device.

Fortunately there are a few hardware devices that do support ssh / gpg key storage - check out Trezor and Jade

This site shows you more devices - just filter on 'other features' and 'hardware based ssh/gpg' - https://thebitcoinhole.com