I woke up this morning to find that one of my channels had co-op closed a channel that I had opened with them and that it cleared at a fee rate of 310 sat/vB, which cost me about 60K sats!
my lnd.conf was set to coop-close-target-confs=6 (a 1 hour target) as per default
Make sure in a high fee environment, you have double checked all of your settings for your own cost/risk profile.
They don't mention coop-close-target-confs but more good things to check here: https://www.lightningnode.info/technicals/highonchainfees
Another thought, it would be great if there were a standard in lightning for nodes to publish their own policies for auto-closing channels. My channel appeared healthy to my own metrics, but with more liquidity on my side. I'm sure they had a policy of something like "if over 80% remote balance for x time, coop close" but given that that information is opaque to me (hidden in their own automation logic), I didn't have an opportunity to adjust the channel to avoid automated coop close...
reply
The logic you'd love to see communicated automatically is too complex. In addition, there are always other reasons (change of mind, drunk operators, ...) that don't fit into this. I, personally, prefer the rating system implemented in lnd-manageJ (see https://github.com/C-Otto/lnd-manageJ/blob/main/rating.md), but that's not the only information I consider.
reply
I'd consider your script a well published set of guidelines that I could reverse into a script to run against my own node to see what score you would give me. So maybe I'm not looking for a lightning standard so much as a best-practice for operators to publish their general standards/scoring methods. IMHO, the best metrics are ones that can't be gamed without the metric being realized in a positive way so publishing these things is good because it helps to validate that you really do have a good rule set rather than an arbitrary hack of drunk decisions ;)
reply
This happened to me today on my LND node, with the CPFP tx at 620 sat/vB. (reported as 2x overpayed). The force close cost almost 6x the value of the channel recovered. This channel was not even created by me. Someone established a 1M sat channel to me, and caused my node to burn 280k in fees on force close and HTLC timeout.
The CPFP transaction cost ~200k sat, with total fees being close ~280k. Total amount recovered in local balance was 50k.
This all seems broken in this environment, and is highly discouraging as a node operator. I have never had such issues with CLN.
reply
Wait… doesn’t the channel opener pay for a force close. Did you manually initiate a CPFP? I’m puzzled as to how this could have come from your pocket.
reply
Yes -- my understanding was the channel opener paid for the close, I did not initiate the CPFP manually. There were 3 tx's, and the final one was claiming an HTLC timeout. It swept the hot wallet for the CPFP, then it swept it again to claim the HTLC timeout. I can share the tx's if you wish.
All this happened about 4 hours ago, without any action from me specifically.
I did not think this could happen either, but in my experience LND just burns sats...
reply
Running a lightning node is certainly a way to spend Bitcoin.
What you are describing almost sounds like a watchtower penalty. I wonder if something got FUBAR on your node and it sent an older state in the force close, leading to a punishment txn… but I don’t see anything from today on this list: https://forkmonitor.info/lightning
Or maybe this is a successful attack vector being implemented in the wild… I am curious to see the txns if you don’t mind sharing. Just to be clear, you initiated the force close? Or they did?
It sounds like the force close was initiated while a pending ln txn was running, which is a little sus.
reply
I am not sure who initiated the close. Its not reporting in Thunderhub, just 'closing'. I think when it is closed it reports remote or local close. I did not initiate any action from my side myself.
reply
Ah, I see what happened now. So it looks like someone sent an HTLC through that timed out while one of you was offline. This is IHMO one of the most horrible node operator situations.
If one of the nodes is offline when a pending HTLC expires, instead of being canceled by both nodes cooperatively, it triggers a force close of the channel to resolve the HTLC on chain. If you were the recipient of the incoming transaction, theoretically you would have that HTLC balance as input from the source (prior to expire)—so what got taken from you on chain should be in your LN balance (just as a balance shift incoming from another channel). This is my understanding of it anyway. This hasn’t happened to me yet…
I’ve also heard of people saying that htlcs can just get stuck when nodes are both online and trigger this. Maybe if you are both tor only and your ping time is really long and tor just shat the bed…
reply
Thank you for taking a look! I think this would account for the HTLC expire tx, in that the sats are going to be in another channel.
I am at a loss on why it spent the whole onchain wallet on such high fees? I think the bulk of the sats were spent from just broadcasting the claim tx's. It blew away several small UTXOs, overspending to consolidate them.
Seems wise to have a minimal amount and few UTXO so high fees don't eat up large claims.
reply
That's why I like onchain. I wonder what Satoshi would have thought about the lightning network?
reply
People often say things like, "Satoshi got so much right"--but I think he didn't think through some things, or just assumed that the software would evolve as needed to compensate for scaling issues. Whatever he thought, I'm certain he didn't envision 8 billion people owning their own UTXOs since it would take something like 40 years worth of blocks for that single UTXO claim to process (and that's even with segwit)--and then what? No on-chain system scales by itself--that's the blockchain CAP theorem: Security, Scalability, Decentralization, you get 2/3 onchain, for the 3rd you need a layer 2. Satoshi made the decision that Decentralization and Security were paramount and that Scalability would come later. Some cryptos mistakenly think that Scaling should be one of the 2 on-chain pillars and that's why they will fail, because they sacrificed on-chain Decentralization or Security.
reply
Good points, I agree, thanks.
reply
I'd like to see LN channel agreement that is fixed. For example: These are the fees forever. Partners cannot pull more than 80% to either side.
Anyone know if there's any reason why this can't happen? Can it be gamed?
reply
if both nodes run the same policy, it's all just code on both sides... if you trust the other node to uphold the standards you agree on, then it's doable, but there would need to be some additional kind of HTLC type deal if you wanted to be able to punish someone for ignoring the rules (aside from deciding to just close the channel on them). I think the best rules in lightning are things like ZBF where it's just a detectable setting and your node can be highlighted for having that configuration. There's no punishment for changing your mind aside from losing your label on node index sites for being ZBF. Positive re-enforcement through providing value.
reply
Does anyone know what this actually does?
allow-circular-route
Sounds like it allows or denys circular rebalances. But, does it allow it for the node or other people using your node to rebalance? I assume the former.
reply
allow-circular-route is disabled by default to prevent a peer from using you to move funds both out and then back into themselves. This doesn't prevent you from circular routing someone's rebalance through multiple hops, just that you wouldn't be the single source of rebalancing. This might not sound like an issue, but it can be part of a type of griefing attack where someone opens a connection to your node, then saturates your pending HTLCs with circular rebalances: https://github.com/lightningnetwork/lnd/issues/3771
IMHO, this is not something most small nodes need to worry about and you might have a valid use-case for doing this between two nodes that you own that are connected to each-other. It would be more useful if this feature were updated to be an allow-list of circular-routing friendly peers (so you could selectively enable this use-case but not allow random connectors to do this).
reply
Very helpful. Thank you!
reply
; If true, our node will allow htlc forwards that arrive and depart on the same ; channel.
But I am still unsure.
reply
Thanks for this. Pretty sure 6 blocks in the default.
reply