What you describe is indeed very problematic.

Here's something else that's problematic: if you made your xpub public and ever revealed even one private key, for example thinking that it's safe because you already spent the money and there's nothing "on" that private key any more, then you accidentally revealed all the private keys in that entire account... (this is warned about in BIP32 btw).

(If you have a multi-account wallet, then at least compromise of one account like this doesn't compromise others .. unfortunately almost nobody uses multi-account wallets any more, though).