pull down to refresh
reply
This was a nightmare. When these migrations fail, you have no idea how long it might take to resolve. Even with tons of QA, we experienced a bunch of problems.
Modeling data for long term storage and millisecond access is hard. Getting the design wrong, like we did with wallets v1, is very expensive.
reply
Yep, we expected a backwards incompatible rolling deployment (db gets migrated, but client doesn't know yet), but this migration was also non-atomic (spread across two migrations) and the second one failed unexpectedly requiring manual intervention.
I woke up at 5a this morning and ek and I were in a video call anticipating something like this. By far, this is the most complicated migration we've done to date. We migrated 'polymorphic' relations (ie a hack with triggers for fast polymorphic joins) and encrypted columns, nearly doubling the number of tables we store wallet related stuff in.