107
votes
We're so back
Had to spend my whole day without refreshing Tildes every 5 minutes 😔 My browser already renamed the Tildes link on my new tab page to "502 Bad Gateway"
Had to spend my whole day without refreshing Tildes every 5 minutes 😔 My browser already renamed the Tildes link on my new tab page to "502 Bad Gateway"
Would be cool to get a postmortem, I can imagine some small config change breaking things haha
It looks like the other server where I upload database backups ran out of space, which was causing uploads for new backups to fail. My backup script creates a new backup, uploads it to the other server, then after the uploads finish it deletes old backups from the actual server hosting the site. But since the uploads were failing, it was never reaching the part of the process to delete the old ones, so then the main server ended up running out of space as well due to keeping too many old backups.
Shouldn't be difficult to fix, I'll make some changes to the backup process tomorrow.
Thanks for everything you do to keep this place running Deimos!
Appreciate the response, and thank you for getting the site back online!
Haven't been around much recently, but still: I appreciate all the effort you put in. 🙏
I’m not going to lie I thought the project was dead. I’m so happy to be proven wrong. Thank you for keeping this little light house alive in a sea of literal bullshit. You are the best.
If your hosts aren't using LVM, it may be worth looking into. Using it and partitioning out my various root directories has saved me more than once to out of storage problems.
YAY! I had anxiety that Tildes was maybe gone forever for reasons I'd never get to know.
It must have been my fault! I refreshed the main page early this morning and it looked fine, then I clicked a comment link 2 seconds later and it died. :(
Ah yes, the Cursed Comment of Death and Destruction. Classic mistake. I wonder why they added that to the site.
The hosting plan ran out of comment click credits, an obscure balance that no one knew existed and took a while to figure out how to replenish the quota.
Still doesn't beat that time the site broke because we had a thread about software that misinterprets the text "null" as an actual null value, and then we found out Tildes had that exact bug.
I always cheer when I see the "You broke Reddit" screen, hoping that it's permanent. No such luck yet. But I'd be sad if I thought I'd permanently broken Tildes.
How dare you. Please be more careful commenting next time.
In my defense nobody told me not to feed tildes after midnight!
You probably put water on it, too. Dont you remember what happened the first time you tried that!?
So that's how I got all those invites..
It's cool, I sent an email to contact@tildes.net and it came back up.
Relieved to see it back. It was down last night before I went to bed, can't remember when I first noticed but it was definitely over 12 hours ago.
I wonder if this was related to the HTTP2 bomb bug that got dropped by those irresponsible security researchers that was present in NGINX
I just woke up so I'll have to find the exact report on this after I get my kids fed.
Edit: https://nvd.nist.gov/vuln/detail/CVE-2026-49975
I was wondering during the outage, is there a secondary discussion channel where one can get updates in such cases?
There's an (official? semi-official? completely unofficial?) discord server. But hardly anyone ever goes there, certainly no official people or notifications that I've seen.
I caused the outage by sympathetic magic. I was about to send my very first invite code (in 8 years on Tildes...) to someone I thought would be simpatico (an extremely cool person) and in need of a supportive community... and 502 nginx all day.
It was so bad it took down my entire home network and I had to run new cabling for my router. I spent 4 hours bringing everything back up and am currently eating Taco Bell because I was too worn out to cook.
It was so tough I actually reached out to @definitelynotafae on bluesky 🫠
We clearly need a back-up Tildes for emergencies.
We have one, it's the tildes minecraft server
I always look at the subreddit but no one posts there lol
Cfabbro posts invited on one sub but they're on a cruise
We made it through together.
You could have hopped over to our sleepy LFG server, a few of us checked in to see if we were experiencing the same thing :)
Yeah that was me. I basically forget this server existed, except today.
Which is unfortunate because when this group was started I got some games in with a couple of folks. Not sure why I wasn't more active.
Couldn't find it when it was down!
Should’ve been more clear, @Sunbutt23 is already a member. But in case you ever want to pop by, here’s an invite link!
It is VERY sleepy, mind. Like basically dead. But in this one instance it was helpful!
Hah, me and a few others were converging on an old Reddit post Cfabbro made a couple months back.
I heard that Tildes was filming car commercials…
…in Japan.
Sorry guys, I used my One Wish Willow to have 0 toxicity on Tildes.
WE ARE SO BACK
hiiii :3
Whew wow. I mean wow. How long was that? Do you know where your children are??
I have children? Why didn't anyone tell me?!
Post that nginx config maybe we can help!