pull down to refresh

During a database backup last night, the filesystem where we store temporary (non-critical) database files got borked somehow. For some reason, the database's cope, likely recreating the temporary files over and over again without anywhere to store and reuse them, caused it to get really slow. It took us awhile to figure out that was the cause and it was fixed by literally turning it on and off again.
Thank you to everyone that reported it, and sorry for the downtime and the verbose error messages. (It's not wholly intentional to show you the sausage making but it helps us diagnose issues easier until we create some kind of error digest system.)
It's so awesome that we get to hear about this directly from the guy who maintains it. You guys rock. No apology necessary. Thank you for all your work. Keep it up!
reply
Yes, agreed. I saw in the saloon thread there were others with the same slowness issue and I could wait patiently knowing k00b would come to the rescue and give a prompt update. Pretty decent UX.
reply
Tech shit could happen all the time. IT guys knows that. Understandable and we appreciate your effort to put it back in time. No need to apologize, man. This show us that SN is getting serious. Happy that is not more serious.
reply
This is a reminder that SN runs on only 1 server. In before SN is yuge
reply
I was wondering if it was only me. Thank you for the debrief!
reply
All goody sir, appreciate the transparency and dedication to us compulsory stackers!
reply
So often the rebooting or recycling seems to fix vexing problems. I have also found that sometimes the simplest correction to be the one that works. You must be using PostGre. Sometimes a beast. i have had good luck with MySQL before.
reply
You could try duplicating your database files, setting up another area and run MySQL on the data to see which runs it better for your usage. Then make the new data fork into both databases and run them in parallel to see which is better.
reply
Glad to hear you used my favorite technique for solving puzzling technical issues.
reply
Show me Devs who respond? Devs that come back to you with an explanation of the problem and what they did to fix it?
Anyone? No? Yeah didn't think so.
This is why these guys are diamonds.
Y'all can have a brofist from Pub 👊
reply
22 sats \ 1 reply \ @ek 30 Nov
Devs that come back to you with an explanation of the problem and what they did to fix it?
To be fair, I would say it looks like most devs don't do that since there's no easy way for them to reach most of their users and vice versa. We're in a unique position where what we're building is a very natural way to interact with our users and for them to interact with us.
You can hear more about this in this panel at minute 17:
Keyan's cheat code is that he built the ultimate focus group.
The feedback loop is incredible on Stacker News. You built the platform that you get users' feedback about the platform
I think there's something generic there, your product should have a mechanism for your users to be able to nag the crap out of you all the time because that's telling you about the reality of them
For full context, watch from 15:28.
reply
the wild west ate @TonyGiorgio
reply
89 sats \ 4 replies \ @nym 29 Nov
No problem glad you got it fixed
reply
Hey, I see you less active these days on SN?
reply
reply
Thanks ek. I wonder how do you remember so much? lol
reply
0 sats \ 0 replies \ @nym 29 Nov
Hey there! Yea just been busy with life and away from the keyboard more often.
reply
Haha Simple solutions. Always nice.
reply
Thank you. It feels the speed and performance has improved from what we had before we had the issue.
There's no need of apology..it can happen.
reply
I was worried!
reply
24 sats \ 0 replies \ @ek 29 Nov
AWS asserted its dominance and tested our incident response
reply
@k00b always grateful for what you tech wizards have built
reply
Thankfully I didn't spend all my fiat on Black Friday Sales while SN was down. I still had enough left to do my daily bitcoin DCA. Phew.
Appreciate your effort on a holiday weekend. I know it's part of the deal when you run the show but still worth our appreciation.
reply
Unfortunately, the A bitcoin experiment post has been loading slower and slower. Haha.
reply
Hat off, sir! First time in 6 months I have noticed a small issue and got it sorted fast. You do a great job! Thanks for all you do.
reply
It's cool, these things happen. A simple reboot fixed it, just like I do to my computer all the time. I was only out of SN for 20 minutes.
reply
Glad it’s fixed now. Thanks for the update! sounds like the database just needed a little nap
reply
Have you tried turning it off and on again?
reply
🫡
reply
So in the end was just this?
reply
Gotta love the power cycle
reply
Thank you for keeping us posted.
reply
I work in IT. It happens. Forward!
reply
reply
No worries! We are still early!
reply
Thank you for your courage.
reply