Found a hack that reduces the cost from 548089.02 -> 5193.26, 100x improvement ... still need to test, but hopefully that will let us grow another couple months without overhauling notifications.
fascinating to read... there are probably papers out in obscure journals about how to optimize this... i'm amazed how responsive twitter notifications are as such a large scale...
reply
They denormalize them most certainly (lots of data duplication in the data models) which is fast but relatively complex to maintain.
reply