If you use a passphrase, you get into a completely new wallet. So yes, your xpubs will change.

No, you can't access someone's wallet by knowing their public key!