No, you should worry when someone get unauthorized access to your private key instead