What is special about the ECDH is that your peer's pubkey defines the curve being used, not your secret key!

So if an attacker provides a pubkey from a weaker curve, and you respond with the product of their weak pubkey * your secret, they can use brute-force to factor out and reveal your secret.