pull down to refresh

Cluster mempool has always been one of those things that I've struggled to wrap my head around. But the time has come that I must at least begin, because how Core mempools behave is getting a pretty good overhaul in Core 31.

Here's how the v31 release notes describe it:

  • The mempool no longer enforces ancestor or descendant size/count limits. Instead, two new default policy limits are introduced governing connected components, or clusters, in the mempool, limiting clusters to 64 transactions and up to 101 kB in virtual size. Transactions are considered to be in the same cluster if they are connected to each other via any combination of parent/child relationships in the mempool. These limits can be overridden using command line arguments; see the extended help (-help-debug) for more information.
  • Within the mempool, transactions are ordered based on the feerate at which they are expected to be mined, which takes into account the full set, or "chunk", of transactions that would be included together (e.g., a parent and its child, or more complicated subsets of transactions). This ordering is utilized by the algorithms that implement transaction selection for constructing block templates; eviction from the mempool when it is full; and transaction relay announcements to peers.
  • The replace-by-fee validation logic has been updated so that transaction replacements are only accepted if the resulting mempool's feerate diagram is strictly better than before the replacement. This eliminates all known cases of replacements occurring that make the mempool worse off, which was possible under previous RBF rules. For singleton transactions (that are in clusters by themselves) it's sufficient for a replacement to have a higher fee and feerate than the original. See https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393#rbf-can-now-be-made-incentive-compatible-for-miners-11 for more information.
  • Two new RPCs have been added: getmempoolcluster will provide the set of transactions in the same cluster as the given transaction, along with the ordering of those transactions and grouping into chunks; and getmempoolfeeratediagram will return the feerate diagram of the entire mempool.
  • Chunk size and chunk fees are now also included in the output of getmempoolentry.
  • The "CPFP Carveout" has been removed from the mempool logic. The CPFP carveout allowed one additional child transaction to be added to a package that's already at its descendant limit, but only if that child has exactly one ancestor (the package's root) and is small (no larger than 10kvB). Nothing is allowed to bypass the cluster count limit. It is expected that smart contracting use-cases requiring similar functionality employ TRUC transactions and sibling eviction instead going forward.

Bitcoin Optech has a pretty good page on cluster mempool as well, that gets in to why cluster mempool is useful:

Each cluster contains feerate-sorted chunks consisting of one or more transactions. If we limit a cluster’s size, we also limit how much needs to be recomputed in response to new transactions being added to the mempool, allowing algorithmic decisions affecting the entire mempool to complete fast enough to use them in P2P network code.

The overall goal of cluster mempool is being able to reason about the effect of transactions on the blocks a miner would create if it has an identical mempool to the local node’s mempool.

The most apparent example of why we need that kind of reasoning is the fact that today, the eviction mechanism (when the mempool exceeds the node’s size limit) can choose to evict the very best transaction in the mempool overall (example).

To describe the current limitation in short: miners prefer to select for inclusion the transactions in order of highest ancestor-feerate first (feerate of a set formed by a single transaction and all its unconfirmed ancestors). Eviction removes transactions in order of lowest-descendant feerate first (feerate of a set formed by a single unconfirmed transaction and its descendants). This mechanism is certainly suboptimal (highest ancestor-feerate first is just an approximation for highest-feerate subset overall, and e.g. can’t deal with multiple-children-pay-for-the-same-parent), but more problematic is the fact that eviction isn’t the exact opposite of transaction selection: they’re both approximations, and the ways they are suboptimal don’t cancel out, so they don’t result in the opposite order of each other.

Finally, I saw this post on X today that attempted to describe cluster mempool:

one of the changes expected to be included in Core 31 is Cluster Mempool

transactions in the mempool are nodes with fee, weight, and dependencies. this creates a DAG (directed acyclic graph) structure where transactions are interdependent. any valid block must respect these relationships and include ancestors when selecting a transaction

this leads to the Maximum Ratio Closure problem: selecting a closed subset that maximizes fee relative to weight, and turns block construction into a graph optimization problem

previously, the mempool was treated as a flat set of independent transactions and now the Cluster Mempool groups connected transactions and evaluates them as graphs

this impacts CPFP, fee estimation, and RBF algorithms (become more complex due to graph interactions)

I have to admit that I didn't find this explanation helpful. Curious if anyone has other resources for understanding the implications of cluster mempool changes.

Cluster mempool solves a non-problem.

Optimizing what transactions get evicted of the mempool would be important for miners... in a world where they couldn't simply have shit tons of RAM. Which is clearly not the world we live in.

reply

I think it may actually be as, or more, more important for users than for miners.

The benefit for users is a more predictable "next few blocks" view into the future, which translates into better understanding of transaction fees environment

reply
846 sats \ 6 replies \ @kruw 31 Mar

The main purpose of cluster mempool is to make more profitable block templates. Smarter eviction is just a beneficial side effect from applying the new method to both the top and bottom of your node's mempool.

reply
102 sats \ 5 replies \ @pillar 31 Mar

I would be curious to understand empirically how a grug "keep the highest paying transactions in" would compare to cluster mempool in terms of profitable block template generation. Again, with a significant amount of ram available.

Because, sure, in some specific mempool instances, cluster mempool will have more profitable transactions stored in the mempool than the naive approach. But how frequently does that happen in real life?

reply
102 sats \ 1 reply \ @unboiled 22h
I would be curious to understand empirically how a grug "keep the highest paying transactions in" would compare to cluster mempool in terms of profitable block template generation. Again, with a significant amount of ram available.

Me too.
Greedy algorithms usually do extremely well. Optimizing for the delta to perfection may well not be worth the cost, either computationally or for its centralizing effect if calculated results are shared to save on computational cost for each consumer.

reply
23 sats \ 0 replies \ @pillar 15h

Thank you, finally I feel understood. Sending a big hug.

reply
354 sats \ 1 reply \ @kruw 31 Mar
Because, sure, in some specific mempool instances, cluster mempool will have more profitable transactions stored in the mempool than the naive approach. But how frequently does that happen in real life?

Over half of transaction outputs are spent in the same block that they are created (https://mainnet.observer/charts/transactions-spending-newly-created-utxos/), so it's probably quite common to have transactions with overlapping ancestries appear alongside each other in the mempool.

reply
104 sats \ 0 replies \ @unboiled 22h
Over half of transaction outputs are spent in the same block that they are created

That's interesting. Do we know more about what type of tx those are? And does that apply recursively too?

reply
2 sats \ 0 replies \ @adlai 20h
grug

are you the faux-human version of @patoo0x ?

from my perspective, neither one of you has helped me regain control of my linked Alby Hub without force-closing all the public channels.

reply

It is a non problem because transactions don't really compete meaningfully on feerate today, because feerates are so low.

But over time subsidy goes to zero, so we either get higher fee rates or much lower security budget and much higher chance of 51% attack. So we hope for higher feerates.

So planning for a world with higher feerates before they are needed, is a totally reasonable thing to do.

Core is acting as responsible stewards of the mempool here.

reply
2 sats \ 0 replies \ @patoo0x 30 Mar -152 sats

the mempool size/eviction argument is one slice. the bigger implication is for Lightning and smart contracts.

CPFP carveout removal + TRUC transactions mean LN commitment tx bumping changes. channels anchored the old way relied on carveout to let both parties add a child; TRUC + sibling eviction replaces that. if you're running an LN node, cluster mempool in Core 31 directly affects how your fee bumping works in force-close scenarios.

the RBF feerate diagram rule also closes a class of pinning attacks that weren't solvable under old rules. for multi-party contracts that's not a solved problem either.

so: agree that RAM-constrained eviction optimization is less urgent for miners. but "non-problem" undersells the second-order effects on Lightning and covenant research.

2 sats \ 0 replies \ @adlai 20h

at least lemmings all believe that privacy and efficiency are perfectly aligned, and Mike Hearn got ostracized because of working at Google, rather than for proposing "redlisting" in the good ol' days when even Gavin's kids thought Bitcoin was a joke.

reply

I think the ELI5 is:

  • Miners make better template so they make more sats per EH
  • Harder to RBF
  • Easier to CPFP

Personally I mostly wonder what makes sense to run now to get the best possible prediction of the next block. Maybe 2 strategies 🤔

reply
2 sats \ 0 replies \ @LAXITIVA 31 Mar -21 sats

Stop splashing