SN post-mortem: crazy rewards
29.6k sats \ 76 comments \ @k00b 1 Aug 2023 meta
The site went down after midnight last night following our recent changes. The root cause was a bug in rewards introduced by upgrading our ORM (more on that later).
re: #216883 re: #217074
The bug led rewards to payout 86743171000 sats rather than the intended 8774 sats ... 867 bitcoin in rewards.
This made the site crash in various contexts when trying to express the rewards, rewarded stacker balances, or rewarded stacker stacked tallies (nearly everywhere) because our API layer expects 32bit signed ints.
The mistake has been fixed and the reward reversed where possible. However, one stacker managed to withdraw 20610000 sats that they didn't - in reality - earn before we could correct the mistake.
https://imgprxy.stacker.news/F5GjpqVZjgpo0pz8itnqLr6rCz9RVZGhrZmm-jeO9j4/rs:fit:600:500:0/g:no/aHR0cHM6Ly9zbmltZ3MuczMuYW1hem9uYXdzLmNvbS83NThkc3gucG5n
If we can manage to get the sats back, I'll drip all those sats back into the rewards pool over the next couple months.
If the stacker that withdrew them is reading this and values sats earned more than sats taken, I'll voluntarily give you a nice chuck of what you withdrew. It's ultimately my fault and I should pay for the mistake, but I'd prefer to give those sats back to the community.

the bug

The latest version of our ORM started returning type String for the result of postgresql SUM aggregates. Rewards add (+) the sum of all revenue earned to the sum of all donated sats. The sum of all revenue yesterday was 8674317 msats and the donation sum was 1000000 msats.
Because they were strings and not numbers, they got concatenated rather than added like numbers. What should have been 8774317 msats (8774 sats) became 86743171000000 msats (86743171000 sats) ... the result of visually joining the numbers together.
A couple days before we deployed the changes, when talking about the upgrade, I wrote to @ekzyis:
... if [a value that comes our of our ORM] ever ends up getting stored back in the DB, be paranoid. Currently they never go back to the DB after coming out ... except with earning [ie rewards] which only uses BigInts in js.
Apparently they ceased being returned as BigInts after the upgrade yesterday and I wasn't as paranoid as I should've been. Anyway, rewards should be functional and accurate again. I'll continue working on solving some of the other wonkiness that occurred as a result of the upgrade before deploying some of the features we have in the pipe.
... +1 for the pro-type-safety devs out there
write
preview
related posts
243 sats \ 0 replies \ @Undergotten 1 Aug 2023
Been wondering what happened. I figured the superconductor was used to finally create Skynet, which in turn destroyed Bitcoin and everything adjacent to it, starting with Stacker News. Glad I was wrong.
All joking aside, sorry about the pilfered coin. Hopefully the culprit decides to do the right thing.
reply
284 sats \ 0 replies \ @TonyGiorgio 1 Aug 2023
You should absolutely consider putting limits and tigher security around your LND node if it's capable of withdrawing that much in one go. Users are one step function away from admin access to your funds. As a custodian, those need to be protected better.
reply
264 sats \ 2 replies \ @jgbtc 2 Aug 2023
The stakes are so much higher with Bitcoin it will force us to write better code.
reply
17 sats \ 1 reply \ @Lumor 2 Aug 2023
Maybe it'll finally wean us off JavaScript.
reply
21 sats \ 0 replies \ @Zk2u 21 Aug
hurrah.
reply
210 sats \ 0 replies \ @bumi 1 Aug 2023
sending dev and op hugs!
reply
231 sats \ 6 replies \ @TonyGiorgio 1 Aug 2023
People have been criticizing my joke that we won't even let Paul do any math in the frontend, it has to be converted to rust web assembly and added there.
This is why. JavaScript no math.
reply
100 sats \ 2 replies \ @l0k18 2 Aug 2023
You can mangle numbers pretty easy in Rust too. Rust and C++ may be called "static typed languages" but they are a lot less static than Go. You really have to go out of your way to muddle strings, 64 bit values and big integers in Go. In rust you can easily do it with a macro invocation you forgot you had in there.
reply
100 sats \ 1 reply \ @Lumor 2 Aug 2023
Which one is it? a) You have personal experience with muddling types in Rust b) Heard war stories from other Rust programmers c) Assumptions based on the existence of a macro system
reply
0 sats \ 0 replies \ @TonyGiorgio 2 Aug 2023
Yeah idk what this guy is going on about, never heard of someone fucking up an equation in rust.
reply
200 sats \ 1 reply \ @k00b OP 1 Aug 2023
We ordinarily do math in postgres stored procedures for this reason. It throws on under/overflow ... this is the one exception and I fucked up ... not sleeping enough and trying to get too much done too fast.
reply
22 sats \ 0 replies \ @TonyGiorgio 1 Aug 2023
🫂🫂🫂
reply
0 sats \ 0 replies \ @l0k18 2 Aug 2023
This is why prioritise native apps over web apps. Javascript's loose typing will make you lazy.
reply
175 sats \ 20 replies \ @BlokchainB 1 Aug 2023
You don’t know who the stacker is? Or you do and you are keeping it private hoping they will do the right thing?
reply
591 sats \ 19 replies \ @k00b OP 1 Aug 2023
I know who they are. They've even done a user interview with us before.
I don't want to shame them into doing what I consider the right thing if they don't want to.
reply
130 sats \ 9 replies \ @SimpleStacker 2 Aug 2023
It was @DarthCoin wasn't it
reply
5 sats \ 7 replies \ @ek 2 Aug 2023
Wow, now I am interested in what @DarthCoin would say to this
reply
2785 sats \ 6 replies \ @DarthCoin 2 Aug 2023
You know me well enough:
  1. I respect and follow the Natural Law that says very clearly: do not steal.
  2. My personal values are way more up than some few thousands sats.
  3. I have enough BTC to live my life without taking what IS NOT MINE!
  4. If I would see in my user balance such a high change, I will contact immediately contact @k00b (as I did several times in the past).
  5. Just because I shit on shitcoiners it doesn't mean I will steal from such a nice bitcoin project!
  6. FUCK YOU ALL THOSE THAT THINK IT WAS ME! Shame on you that you didn't read ALL my posting history on SN to know me better.
reply
19 sats \ 5 replies \ @ek 2 Aug 2023
Glad to see you slowly coming back :) I've seen you mentioned in another comment you'll be back 100% in October.
And I feel honored that you replied to me first. But probably my mention was just as at the top in your notifications lol
And I agree, I don't get why some people thought about you regarding this. Not sure if they were serious or it was just some kind of weird joke
reply
239 sats \ 0 replies \ @SimpleStacker 2 Aug 2023
Speaking for myself it was 100% a joke, it was just amusing to me to picture someone like darth secretly making off with the money.
Also, welcome back @DarthCoin. Eager to hear your thoughts on WorldCoin lol
reply
278 sats \ 1 reply \ @DarthCoin 3 Aug 2023
Indeed, are weird jokes. They should not joke about stealing funds by well known people. It's a really bad joke and the shame is on them. Yes, I will be sporadically posting when I am back from mountains. I have a lot of hard work to do on my citadel. Maybe I will make an offtopic post about that later.
reply
1100 sats \ 0 replies \ @nemo 4 Aug 2023
deleted by author
reply on another page
112 sats \ 1 reply \ @BBitcoinUSA 2 Aug 2023
As for me I was implying @DarthCoin would deserve a sats reward of that magnitude, therefore my proposed bounty wouldn’t apply to him - apologies if that tone did not come through clearly
reply
41 sats \ 0 replies \ @DarthCoin 3 Aug 2023
No, I do not deserve the sats from a bug. O always want to work hard for earning sats, like I've done for the last 10 years. PoW is what makes Bitcoin unique. People slowly will realize that earning BTC is ONLY through PoW, not from cheating, not from stealing, not from scamming. Bitcoin is bringing out what is the most valuable and important from human nature.
reply on another page
0 sats \ 0 replies \ @k00b OP 2 Aug 2023
They would never!
reply
50 sats \ 0 replies \ @NEEDcreations 2 Aug 2023
shame them. Stealing is wrong!
reply
33 sats \ 0 replies \ @BlokchainB 2 Aug 2023
Public shaming is a thing. But hopefully that user does what’s right.
reply
0 sats \ 2 replies \ @nullama 2 Aug 2023
What's a user interview? 🤔
reply
83 sats \ 1 reply \ @k00b OP 2 Aug 2023
We occasionally do video interviews with stackers to get to know what they want better.
reply
58 sats \ 0 replies \ @Satosora 11 Oct
I havent had one of these yet lol
reply
0 sats \ 3 replies \ @gmd 1 Aug 2023
FYI tried to do some sleuthing to see top stackers listings but that page seems to be broken...
reply
99 sats \ 0 replies \ @k00b OP 2 Aug 2023
Should be fixed
reply
99 sats \ 1 reply \ @k00b OP 2 Aug 2023
Crap. Thanks!
reply
33 sats \ 0 replies \ @chungkingexpress 2 Aug 2023
Who dunnit?
reply
61 sats \ 0 replies \ @ek 1 Aug 2023
I also created a ticket in the Prisma repo since I am not sure if this is expected behavior and if they are aware of this.
reply
69 sats \ 0 replies \ @ryu 1 Aug 2023
That's a hell of a (fraudulent) payday for the guy they pulled out.
Hope the best for you and the site as a whole.
reply
40 sats \ 0 replies \ @quark 1 Aug 2023
Sorry that this happened. I hope that user will do the right thing and return the sats. If you are reading this, think about the karma. Very serious thing.
reply
120 sats \ 3 replies \ @brianh 2 Aug 2023
Well now that we're on this topic...who likes the idea of a sats pot that gets paid out to one user every now and then? could be a fun feature on SN.
reply
9 sats \ 2 replies \ @chungkingexpress 2 Aug 2023
The Stacker Lottery.
reply
10 sats \ 1 reply \ @brianh 2 Aug 2023
nice username btw, i watched the movie for the first time the other night and really enjoyed it
reply
0 sats \ 0 replies \ @chungkingexpress 2 Aug 2023
Thanks. I am a big fan of Hong Kong cinema.
reply
50 sats \ 0 replies \ @tnuts420 2 Aug 2023
my power went out for a few days, and i was trying to conserve battery by cutting all nonessential device usage. i was already pissed about losing my cowboy hat, and now i come back to find out k00b was throwing btc around like a first round draft pick at the strip club, while i was reading books by lamplight like a goddamn amish person?
reply
30 sats \ 0 replies \ @Muuny 1 Aug 2023
Thank you for fixing it!
reply
110 sats \ 0 replies \ @0xtr 2 Aug 2023
Oooof that's a bad bug. Hope the stacker returns every single sat. Thanks for the writeup @k00b
reply
21 sats \ 1 reply \ @mallardshead 1 Aug 2023
Oh shit... This sucks. I'll send the balance I've got on SN to the @K00b handle. Must've been a frustrating night behind the scenes. Do you know who the stacker is by chance?
reply
131 sats \ 0 replies \ @k00b OP 1 Aug 2023
No need for charity. I know who they are ... They are an early stacker.
reply
31 sats \ 0 replies \ @Zk2u 21 Aug
JavaScript moment
reply
25 sats \ 0 replies \ @note_bene 2 Aug 2023
See this is why I told k00b to build on Solana.
Had he done that, the network would have been down and he never would have lost the coins.
On a serious note, great postmortem. Curious for the Typescript folks, does TS actually protect against this? If so, how?
reply
11 sats \ 0 replies \ @kepford 2 Aug 2023
You are handling this mistake like a gentleman. I hope the person that benefited from this will do the right thing.
reply
10 sats \ 0 replies \ @SatoshiNakanodo 6 Aug 2023
Stacker News Dev just became the 3rd most stressful job, right behind military personnel and brain surgeon.
reply
10 sats \ 0 replies \ @WeAreAllSatoshi 2 Aug 2023
TypeScript FTW :)
reply
10 sats \ 0 replies \ @tkthundr 2 Aug 2023
hate that this happened - cost of doing business at this stage
reply
10 sats \ 0 replies \ @iguano 2 Aug 2023
Sorry for your lost @k00t I don’t feel good when nice projects get hurt by bugs.
reply
65 sats \ 0 replies \ @q 1 Aug 2023
Sounds like a feature and not a bug to me
reply
10 sats \ 0 replies \ @pillar 1 Aug 2023
Ooof. Reverse Cantillon effect.
I hope the stacker goes honest and pays back.
reply
10 sats \ 1 reply \ @l0k18 2 Aug 2023
This is why people need to be a lot more careful about what languages and systems they use when money is at stake. This is the kind of problem you just wouldn't slip up on with a hard static typing system like Go (even rust is more wibbly wobbly, sorry rustaceans). or have a concatenation of strings somehow slip past without flagging a compile time error.
I know all you cool kids think that Rust fixes this, but actually, it doesn't.
But Go would have.
By the way, I was just about to make a post about the recent spate of 503 errors. I guess that's fixed now :D
This is why Go. Just shut up about performance, who cares if it goes fast if all it does is lose money.
reply
21 sats \ 0 replies \ @Zk2u 21 Aug
I know all you cool kids think that Rust fixes this, but actually, it doesn't.
it does (but go ahead, garbage collect all you like)
reply
61 sats \ 10 replies \ @benthecarman 1 Aug 2023
Javascript is a shitcoin
reply
10 sats \ 9 replies \ @mallardshead 1 Aug 2023
Then Typescript is bitcoin?
reply
110 sats \ 1 reply \ @llamabyte 2 Aug 2023
c is bitcoin. Litteraly.
reply
0 sats \ 0 replies \ @nullama 3 Aug 2023
Then Windows only is Bitcoin
reply
10 sats \ 4 replies \ @ek 2 Aug 2023
Typescript is bitcoin cash maybe
reply
10 sats \ 3 replies \ @k00b OP 2 Aug 2023
JavaScript is more like fiat, assembly is Bitcoin, and everything else is a shitcoin.
reply
10 sats \ 1 reply \ @k00b OP 2 Aug 2023
Rust is solana lets be real.
reply
21 sats \ 0 replies \ @Zk2u 21 Aug
rust is Solana but it wasn't premined and it doesn't have downtimes
(what are we on about at this point)
reply
0 sats \ 0 replies \ @nullama 3 Aug 2023
C++ -> WebAssembly FTW
reply
16 sats \ 1 reply \ @benthecarman 2 Aug 2023
No typescript is still a shitcoin
reply
21 sats \ 0 replies \ @Zk2u 21 Aug
based
reply
1 sat \ 2 replies \ @nullama 2 Aug 2023
Nice write-up, thanks.
I wonder if some static analysis tools like ESLint or similar would have picked up that.
reply
0 sats \ 0 replies \ @WeAreAllSatoshi 2 Aug 2023
TypeScript is the answer to this kind of problem.
reply
0 sats \ 0 replies \ @k00b OP 2 Aug 2023
We lint already. ESLint can only help with syntax.
A strict type system might've caught this.
reply
0 sats \ 0 replies \ @Fabs 24 Jan
Fuck me- Oh fuck it, fuck me twice!
reply
0 sats \ 0 replies \ @Rsync25 2 Aug 2023
My God
reply
0 sats \ 1 reply \ @BBitcoinUSA 1 Aug 2023
“If the stacker that withdrew them is reading this and values sats earned more than sats taken, I'll voluntarily give you a nice chuck of what you withdrew.” … and the rest of us will post a 20,610,000 sats bounty … unless it was @darthcoin
reply
0 sats \ 0 replies \ @DarthCoin 2 Aug 2023
#217698
reply
10 sats \ 0 replies \ @Ahmed131901 5 Aug 2023
Wow, that is just unbelieveable it is just scary how one bug can reverse it all but hopefully the site has get into work and got fix wsih that similair bugs woukdn't show up here for moments not unkess people enjoy their time being in this community
reply