pull down to refresh
Yep, we expected a backwards incompatible rolling deployment (db gets migrated, but client doesn't know yet), but this migration was also non-atomic (spread across two migrations) and the second one failed unexpectedly requiring manual intervention.
I woke up at 5a this morning and ek and I were in a video call anticipating something like this. By far, this is the most complicated migration we've done to date. We migrated 'polymorphic' relations (ie a hack with triggers for fast polymorphic joins) and encrypted columns, nearly doubling the number of tables we store wallet related stuff in.
This was a nightmare. When these migrations fail, you have no idea how long it might take to resolve. Even with tons of QA, we experienced a bunch of problems.
Modeling data for long term storage and millisecond access is hard. Getting the design wrong, like we did with wallets v1, is very expensive.
Also, I noticed a brief 500 when loading SN earlier this morning. I assume it was during this deployment?