Your thinking is correct.

Adding a passphrase is just adding a 13th or 25th word to your BIP39 seed phrase.

All the BIP39 seed phrase does is represent your private key using English words instead of binary, hexidecimal, WIF, or even decimal format.

Your private key is just a 256 bit number. So any private key is just as easy/difficult to crack as any other key generated with 256bits of entropy. It does not matter how the key is represented (be it in BIP39, binary, or hex) resistance to brute force is only achieved by increasing the bits of entropy.

Have a look at https://btcpuzzle.info for examples of keys that have been cracked-by-brute due to using low entropy.