This was actually a red herring. While the above is slow, the biggest problem is notifying you about descendant comments, but not notifying you more than once if you have more than one ancestor to a reply.

That is YOU1 > someone1 > YOU2 > ... > someone2 ... When someone2 comments, you only want one notification even though it's the descendant of two of your comments, YOU1 and YOU2. In order to make only one notification, I have to take the list of ALL replies to you, then make them distinct which requires hashing every reply to you ever.

As you become a mature user the issue gets worse and worse which is why my notifications are so slow. I need to figure out a hack that prevents me from having to make these items distinct.