If they have a fitting private key, they can use it to sign the transactions. Hardware wallets house the keys generated via seed phrase generation, etc. This hypothetical person could have used your seed phrase to generate the private key on another hardware wallet. There is no mutual exclusivity between hardware wallets, meaning many can be loaded with the same private key derived from the same seed phrase.

Anyone more knowledgable than me, please correct me if I made any mistakes.