The only downside is the bad actor still can pretend to be you in the "no longer valid" hot wallet
Sure but probably not for very long, word gets round
As soon as he does something bad people will stop following him and say "yeah I don't like so-and-so anymore because of X bad thing" and then someone else will tell them, "Oh, that's not him anymore, his key got compromised, this is his new key: <whatever> -- and btw if your client supported nip-whatever it would have automatically migrated you as a follower to his new key. Use client-whatever or ask your dev to add support"