During a database backup last night, the filesystem where we store temporary (non-critical) database files got borked somehow. For some reason, the database's cope, likely recreating the temporary files over and over again without anywhere to store and reuse them, caused it to get really slow. It took us awhile to figure out that was the cause and it was fixed by literally turning it on and off again.
Thank you to everyone that reported it, and sorry for the downtime and the verbose error messages. (It's not wholly intentional to show you the sausage making but it helps us diagnose issues easier until we create some kind of error digest system.)
Tech shit could happen all the time. IT guys knows that.
Understandable and we appreciate your effort to put it back in time.
No need to apologize, man. This show us that SN is getting serious.
Happy that is not more serious.
This is a reminder that SN runs on only 1 server. In before SN is yuge
It's so awesome that we get to hear about this directly from the guy who maintains it. You guys rock. No apology necessary. Thank you for all your work. Keep it up!
Yes, agreed. I saw in the saloon thread there were others with the same slowness issue and I could wait patiently knowing k00b would come to the rescue and give a prompt update. Pretty decent UX.
Glad to hear you used my favorite technique for solving puzzling technical issues.
All goody sir, appreciate the transparency and dedication to us compulsory stackers!
I was wondering if it was only me. Thank you for the debrief!
No problem glad you got it fixed
Hey, I see you less active these days on SN?
#768387
Thanks ek. I wonder how do you remember so much? lol
Hey there! Yea just been busy with life and away from the keyboard more often.
So often the rebooting or recycling seems to fix vexing problems. I have also found that sometimes the simplest correction to be the one that works. You must be using PostGre. Sometimes a beast. i have had good luck with MySQL before.
You could try duplicating your database files, setting up another area and run MySQL on the data to see which runs it better for your usage. Then make the new data fork into both databases and run them in parallel to see which is better.
Show me Devs who respond? Devs that come back to you with an explanation of the problem and what they did to fix it?
Anyone? No? Yeah didn't think so.
This is why these guys are diamonds.
Y'all can have a brofist from Pub 👊
To be fair, I would say it looks like most devs don't do that since there's no easy way for them to reach most of their users and vice versa. We're in a unique position where what we're building is a very natural way to interact with our users and for them to interact with us.
You can hear more about this in this panel at minute 17:
view on www.youtube.comFor full context, watch from 15:28.
the wild west ate @TonyGiorgio
Haha
Simple solutions.
Always nice.
Thank you. It feels the speed and performance has improved from what we had before we had the issue.
There's no need of apology..it can happen.
@k00b
always grateful for what you tech wizards have built
AWS asserted its dominance and tested our incident response
Thankfully I didn't spend all my fiat on Black Friday Sales while SN was down. I still had enough left to do my daily bitcoin DCA. Phew.
Appreciate your effort on a holiday weekend. I know it's part of the deal when you run the show but still worth our appreciation.
Unfortunately, the A bitcoin experiment post has been loading slower and slower. Haha.
Hat off, sir! First time in 6 months I have noticed a small issue and got it sorted fast. You do a great job! Thanks for all you do.
It's cool, these things happen. A simple reboot fixed it, just like I do to my computer all the time. I was only out of SN for 20 minutes.
I was worried!
Have you tried turning it off and on again?
🫡
So in the end was just this?
Gotta love the power cycle
Thank you for keeping us posted.
I work in IT. It happens. Forward!
No worries! We are still early!
Thank you for your courage.
Glad it’s fixed now. Thanks for the update! sounds like the database just needed a little nap