Physical access will always be the weak point of any hardware wallet.

If it leaves your sight, assume it's compromised and sweep to a new wallet.