You're basically correct. Moving will potentially be expensive if you have taxable accounts that have gains. But if you have non-tax accounts that can be rolled-over, then it's not really that bad (but there'll be paperwork...).

Of course, you could also have accounts at Vanguard you can't move, such as employer plans. In that case, you're just SOL. But in those accounts you're probably hosed anyway as you often can't choose much anyway.