Discord doesn't appear to be affected. I did read somewhere they were dealing with the extra load by disabling non-essential server load (like typing notifications).
Discord doesn't appear to be affected. I did read somewhere they were dealing with the extra load by disabling non-essential server load (like typing notifications).
A relevant article on how other sites aren't taking the current extra load of people well: Twitter, Is It Down, and tons of other sites are struggling due to the Facebook outage This is many...
This is many things that I don't know, but I find it most glaringly a decent demonstration of why centralized social media websites are bad, given even the Oculus store is down.
Oh I am a misanthrope, I don't pretend I am superior buts that's just who I am. I need a messenger that is reliable for emergencies and eventual small things, I guess. I dread calling people, this...
Oh I am a misanthrope, I don't pretend I am superior buts that's just who I am. I need a messenger that is reliable for emergencies and eventual small things, I guess. I dread calling people, this has always caused profound anxiety and prevented me from doing things. I have been like this since way before messaging. Facebook is pretty reliable, no? That's the first major outage I've heard of. I see no reason to change. It's not like my parents would love to learn how to use Signal anyway.
This has been a quest of ours for the better part of a year. The only remaining holdouts are my aunt and a couple of stubborn friends. It really shouldn't be this hard. In the 90s we changed...
This has been a quest of ours for the better part of a year. The only remaining holdouts are my aunt and a couple of stubborn friends. It really shouldn't be this hard. In the 90s we changed messaging platforms like socks
Well I bought 1kg of fresh sardines and only my father knows how to make them the way I like. He only uses Messenger. I am now eating subpar sardines. Thank you, Facebook.
Well I bought 1kg of fresh sardines and only my father knows how to make them the way I like. He only uses Messenger. I am now eating subpar sardines. Thank you, Facebook.
I'm in a similar boat. I got some good news today and I wanted to share it with my mother but I can't because we're only able to communicate via WhatsApp. Shucks.
I'm in a similar boat. I got some good news today and I wanted to share it with my mother but I can't because we're only able to communicate via WhatsApp. Shucks.
I'm not the person you were asking but just as an example of how this might happen - my mum does have a phone, but she also has terrible mobile reception in her house. She has good wifi though, so...
I'm not the person you were asking but just as an example of how this might happen - my mum does have a phone, but she also has terrible mobile reception in her house. She has good wifi though, so Whatsapp is the most reliable way I can reach her. Calls often fail, texts sometimes take a day or more, especially since covid because she's been going out less (50m up the road from her house she has five bars - there's some geography going on...)
At risk of being one of those guys who throws excess tech at anything vaguely looking like a problem: she might benefit from setting up WiFi calling & SMS. Most UK networks support it, as do a...
At risk of being one of those guys who throws excess tech at anything vaguely looking like a problem: she might benefit from setting up WiFi calling & SMS. Most UK networks support it, as do a significant chunk of phones (that list isn't exhaustive, but it's a great guide to roughly where to look), so it'd be a good emergency backup for situations exactly like yesterday.
[Edit] cc @kwyjibo, in case it's relevant/helpful to you!
Unfortunately my entire family is on GiffGaff, no wifi calling support from them yet, but I'm sure they'll get there eventually. However, it's something I didn't know about and am glad I now do so...
Unfortunately my entire family is on GiffGaff, no wifi calling support from them yet, but I'm sure they'll get there eventually. However, it's something I didn't know about and am glad I now do so thanks for posting.
My Mum's cheap Samsung phone is part of the problem, because my Dad's Pixel 3a is totally fine. In an emergency I can always call him. But he's really awkward to talk to on the phone for some reason.
I am aware of Wi-Fi calling, I use it myself. Unfortunately, my mother's phone doesn't support it. I honestly didn't bother with a backup method because her farm is a 20 minutes drive from where I...
I am aware of Wi-Fi calling, I use it myself. Unfortunately, my mother's phone doesn't support it. I honestly didn't bother with a backup method because her farm is a 20 minutes drive from where I live and there are at least 5 other people on the farm at any given time besides her, so if there is an emergency, I will reach her.
I do appreciate the advice though. I didn't go out of my way to provide her with a backup method, but we'll probably get her a new phone next year so we might as well make sure it's something that supports Wi-Fi calling.
My mum lives in the Scottish highlands. Which, perhaps surprisingly, tend to have pretty good signal in my experience. It really is just one end of the road where she lives that is a dead zone....
My mum lives in the Scottish highlands. Which, perhaps surprisingly, tend to have pretty good signal in my experience. It really is just one end of the road where she lives that is a dead zone. And then only really for her phone, my dad's phone is fine.
Well, she does if she has WhatsApp. I can't speak for the other user, but I live halfway around the world from my parents. Unless it's life or death, I'm not going to be making regular phone calls...
Well, she does if she has WhatsApp. I can't speak for the other user, but I live halfway around the world from my parents. Unless it's life or death, I'm not going to be making regular phone calls to them, I'd just wait for WhatsApp or whatever we use to come back up (we actually use Telegram more).
I would anticipate it being some time until everything is working normally. It's not possible to know for certain, but it's likely from what we know about how complete the failures were that the...
I would anticipate it being some time until everything is working normally. It's not possible to know for certain, but it's likely from what we know about how complete the failures were that the bad route update that excised Facebook from the Internet also completely disrupted their internal networking. Every system which needed to talk over a routed network to another system most likely couldn't. It's rare for services to recover gracefully from that sort of thing, so they will be restarting services and ironing out glitches for some time still.
I am really surprised reading comments of people who rather wait any time it takes for Facebook and WhatsApp go online again than use regular phone calls or SMS. This proved how deep is Facebook...
I am really surprised reading comments of people who rather wait any time it takes for Facebook and WhatsApp go online again than use regular phone calls or SMS. This proved how deep is Facebook and its ecosystem integrated on people lives even knowing how shit Facebook it is.
This is a hard battle not even Telegram can win it and Signal is a lost case.
sms and phone calls don't work for everyone, especially those with international family/friends. And phone calls don't allow video. Not sure why you've given up on Telegram and Signal. I'm slowly...
sms and phone calls don't work for everyone, especially those with international family/friends. And phone calls don't allow video. Not sure why you've given up on Telegram and Signal. I'm slowly converting all my network to signal in the hope of one day deleting whatsapp
I ripped off the bandaid and deleted Whatsapp, leaving anyone who I don’t have on Signal or SMS behind. A few have told me I didn’t answer them on WhatsApp but it’s been relatively painless.
I ripped off the bandaid and deleted Whatsapp, leaving anyone who I don’t have on Signal or SMS behind.
A few have told me I didn’t answer them on WhatsApp but it’s been relatively painless.
I used to live abroad in other continent and phone calls, SMS and email were fine back then. When i do business i still give my phone number and people still call me by phone call, i would like to...
I used to live abroad in other continent and phone calls, SMS and email were fine back then. When i do business i still give my phone number and people still call me by phone call, i would like to know what makes people stop not using phone calls on this rare circumstances. I do use Telegram and WhatsApp btw.
I don't really understand this framing, honestly. Obviously the network effect is an important thing, but I use both Telegram and Signal with non-techy family and friends on the regular. It works...
I don't really understand this framing, honestly. Obviously the network effect is an important thing, but I use both Telegram and Signal with non-techy family and friends on the regular. It works for us. Why does it have to "win" in a global ecosystem? Why can't we just have multiple products, preferably with some level of interoperability?
I need to rephrase a little better, but win i meant a battle versus Facebook ecosystem. Multiple products as messaging apps has always been the case since ICQ.
I need to rephrase a little better, but win i meant a battle versus Facebook ecosystem. Multiple products as messaging apps has always been the case since ICQ.
I've read a few articles saying that this was caused by a DNS failure. I kind of understand that, but what I don't understand is why this is happening. What are the possible reasons for this?
I've read a few articles saying that this was caused by a DNS failure. I kind of understand that, but what I don't understand is why this is happening. What are the possible reasons for this?
Ars have a being-updated article too. Via there, a since-deleted reddit comment from an alleged FB engineer: I haven't messaged my friend who works there because either he's in full panic mode and...
the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
I haven't messaged my friend who works there because either he's in full panic mode and firefighting, or he doesn't know anything which isn't already public. Or possibly both.
Depending on what team they're on its very possible they're just chilling because there's nothing they can do. I'm not going to lie, it's weirdly entertaining being in a sev1 or sev0 incident when...
Depending on what team they're on its very possible they're just chilling because there's nothing they can do.
I'm not going to lie, it's weirdly entertaining being in a sev1 or sev0 incident when it has nothing to do with your team. It's that fly on the wall feeling. Shits going bad, buuut nothing you can do about it.
Agreed, the vast majority of FB are just watching now. Lots of them probably not even that, unless they're working from home. There's a small handful of very sweaty sysadmins somewhere though. I...
Agreed, the vast majority of FB are just watching now. Lots of them probably not even that, unless they're working from home. There's a small handful of very sweaty sysadmins somewhere though. I feel for them. Although it is still fun to watch. I've been on both sides and I definitely prefer the "feet up, can't work even if I wanted to, not my problem" side!
Still too early to tell. Kreb's article is the best you can get for what we know right now. https://krebsonsecurity.com/2021/10/what-happened-to-facebook-instagram-whatsapp/
Still too early to tell. Kreb's article is the best you can get for what we know right now.
Cloudflare's article is a bit more technical, if that's what anyone wants. The super brief summary is that they pulled their BGP routes (this should be sounding very familiar to outage...
The super brief summary is that they pulled their BGP routes (this should be sounding very familiar to outage connoisseurs…), which removed their DNS servers from the Internet, therefore no DNS (but the missing DNS was a symptom, not the root cause).
We will see. It's a fascinatingly bad failure, in a "you can only see the tip of the iceberg" sort of way. I've seen people wondering if it's an attack and though I don't personally think it would...
We will see. It's a fascinatingly bad failure, in a "you can only see the tip of the iceberg" sort of way.
I've seen people wondering if it's an attack and though I don't personally think it would be, it's plausible. Everything is at this point. Hell maybe it's @kfwyre trying to convert more people to screenless day.
I love this headcanon where I'm a ridiculously capable white hat hacker who is able to bring down the biggest tech company in the world in the interest of helping the public at large disrupt the...
I love this headcanon where I'm a ridiculously capable white hat hacker who is able to bring down the biggest tech company in the world in the interest of helping the public at large disrupt the presence of toxic tech in their lives. :)
The reality is much more down to earth. Just today I had to spend >10 minutes in a meeting walking some of my coworkers through how to attach a PDF to an online assignment -- click by excruciating click.
It's an astonishingly bad failure. Especially if the stories about employees not being able to get into FB buildings is true. But even if they're not, someone still managed to take down three of...
It's an astonishingly bad failure. Especially if the stories about employees not being able to get into FB buildings is true. But even if they're not, someone still managed to take down three of the world's biggest internet services and FB's internal comms with, essentially, a single command. They have failsafes in place for so many things at so many levels on their network but this level of fuckupery is amazing.
I feel bad for their admins. Especially whoever pressed enter and then watched it all collapse. I can only imagine the sinking feeling as the entire monitor board started to light up, more and more and more and then ran out of lights and probably just caught fire.
I've had bad sysadmin days. I've had a few very bad sysadmin days that I still occasionally get mild panic attacks about - but nothing like this.
Curious to know if anyone with networking experience knows what needs to be done to fix this? At this point, I'm assuming that any simple "roll it back" fix is not feasible and people are...
Curious to know if anyone with networking experience knows what needs to be done to fix this?
At this point, I'm assuming that any simple "roll it back" fix is not feasible and people are sufficient locked out of fixing this remotely. Is someone basically having to fix this by physically patching each server/router?
Without knowing what’s gone wrong no one could give a reasonable explanation of what’s needed to fix this. A website being down tells you about as much as a car failing to start.
Without knowing what’s gone wrong no one could give a reasonable explanation of what’s needed to fix this. A website being down tells you about as much as a car failing to start.
https://youtu.be/KLqH8p1QVpY Jeff Geerling just put this quick explainer out on what we know so far. The going theory is that the BGP was affected, and it's affecting everything from remote tools...
Jeff Geerling just put this quick explainer out on what we know so far. The going theory is that the BGP was affected, and it's affecting everything from remote tools to door badges. Couple that with less people on site who aren't as specialized, and you have a big old disaster.
This is quite possibly the most predictable, lowest effort comment section I've ever seen on a Tildes submission. And the fact that so many users felt the comments celebrating this were worthy of...
This is quite possibly the most predictable, lowest effort comment section I've ever seen on a Tildes submission. And the fact that so many users felt the comments celebrating this were worthy of a vote, instead of being noise labeled, is incredibly disappointing.
Edit: Well, that turned around quick. Goes to show the power of the noise label, I suppose... when people actually bother to use it anyways.
Personally I'd rather you reply to me directly to express your displeasure than post this kind of comment, but that's just me. Meta commentary like this is just as "noisy" and non-contributing as...
Personally I'd rather you reply to me directly to express your displeasure than post this kind of comment, but that's just me. Meta commentary like this is just as "noisy" and non-contributing as what I posted in my opinion, and I don't think that's controversial to say.
I'll also happily point out that I was the first to point out the suspected technical reasons for this downtime in a since noise-nuked subthread, which is more than you've contributed.
I appreciate your sentiment here but I don't want things to get heated. "which is more than you've contributed" isn't necessary to get your point across.
I appreciate your sentiment here but I don't want things to get heated. "which is more than you've contributed" isn't necessary to get your point across.
I'm still trying to work out what was wrong with your comment exactly. Is voicing an opinion or having a dig at a company frowned upon around here? The majority of the comments seem to be about...
I'm still trying to work out what was wrong with your comment exactly. Is voicing an opinion or having a dig at a company frowned upon around here? The majority of the comments seem to be about the technical side of it anyway, but I cannot personally blame anyone for celebrating the witch being (temporarily) dead, so to speak.
One liners, jokes, quips, "I agree", and others that would reasonably be considered "low effort" are generally frowned upon here. Other users can label such comments as "Joke" or "Noise", which...
One liners, jokes, quips, "I agree", and others that would reasonably be considered "low effort" are generally frowned upon here. Other users can label such comments as "Joke" or "Noise", which under some criteria will auto-collapse that comment (other labels are "Offtopic", "Malice", and "Exemplary"). There's no need to write an essay, but the goal is to have all comments meaningfully contribute to the discussion in some way.
I tagged them all as joke, but felt similarly. There's no resistance to the slow slippery slope into Reddit 2.0, but there's a lot of reasons why it's been headed that way for awhile.
I tagged them all as joke, but felt similarly. There's no resistance to the slow slippery slope into Reddit 2.0, but there's a lot of reasons why it's been headed that way for awhile.
Eh, I feel like we are still a looooong way from even being 2010 Reddit. Having a few low effort one-liners every once in a while isn't so bad as long as there is still thoughtful discussion going...
Eh, I feel like we are still a looooong way from even being 2010 Reddit. Having a few low effort one-liners every once in a while isn't so bad as long as there is still thoughtful discussion going on in most places.
I've been told that Reddit had always been accepting of humor (at least when comments became a thing). It's not like it was HackerNews in its early days. I understand, though, that Tildes had a...
I've been told that Reddit had always been accepting of humor (at least when comments became a thing). It's not like it was HackerNews in its early days.
I understand, though, that Tildes had a different starting off point. I will say it's been interesting to see the progression of this place. Not that I was here in it's early days as I had no idea about this place's existence until a few months ago. But reading through older posts I've noticed that the style of the comments have been changing. I now see a lot more abbreviations being used, specifically "lol." And emoticons are used now as well. That does feel like it clashes with the intended goals of this forum, even aesthetically it feels odd to see someone type out ":P." So I can see the frustration that older users have with the small scale Eternal September that's been going on.
To be absolutely clear I believe that relaxing some of the rules around humor and formality has been good for this place. The frustration is not about that, but the slow erosion of shared values...
To be absolutely clear I believe that relaxing some of the rules around humor and formality has been good for this place. The frustration is not about that, but the slow erosion of shared values on this website which has been repeatedly chipped away by bad actors who regularly flirt with what is considered acceptable. There is only so much resistance that can be offered before it takes too high of an emotional toll on the groups attempting to protect civility and without more active moderation and a set of tools and stronger and regularly enforced guidelines this erosion happens to any platform of scale. Assholes will always win out on nice people because nice people would rather go somewhere else when they aren't treated with respect whereas assholes often do not care if they are disrespected as they don't value others opinions as much as their own.
I don't believe that's necessary. A submission that inspires a bunch of low effort comments can still be newsworthy and worth sharing, as was the case here. I think people might just need to...
I don't believe that's necessary. A submission that inspires a bunch of low effort comments can still be newsworthy and worth sharing, as was the case here. I think people might just need to occasionally be reminded that the noise label exists in order to counteract that trend, as was also the case here too.
It's worse than DNS. The IP address range that points to Facebook's DNS servers no longer does so. It's not something I've personally ever worked with, but this is a BGP problem, apparently.
It's worse than DNS. The IP address range that points to Facebook's DNS servers no longer does so. It's not something I've personally ever worked with, but this is a BGP problem, apparently.
Signal and Telegram are both extremely slow, most likely due to knock-on effects.
Discord doesn't appear to be affected. I did read somewhere they were dealing with the extra load by disabling non-essential server load (like typing notifications).
A relevant article on how other sites aren't taking the current extra load of people well: Twitter, Is It Down, and tons of other sites are struggling due to the Facebook outage
This is many things that I don't know, but I find it most glaringly a decent demonstration of why centralized social media websites are bad, given even the Oculus store is down.
This may be the time to finally kill my family's WhatsApp group chat and move them to Signal.
But alas. . . I have no way to reach them.
You don't have their phone numbers? It might be good to fix that.
Sure I do. But that's the nature of messaging apps. Nobody wants to move to different network that doesn't have all their friends
Not OP but I have everyone's phone number, I just don't wanna talk in real time with them (or almost anyone) :P
There are also text messages, and email (if you have everyone’s email).
Man now the sudden people forgot about phone calls and SMS.
I mean, if you're not willing to put in a little effort to help them along, why should they put in the effort to change to a different messenger app?
Oh I am a misanthrope, I don't pretend I am superior buts that's just who I am. I need a messenger that is reliable for emergencies and eventual small things, I guess. I dread calling people, this has always caused profound anxiety and prevented me from doing things. I have been like this since way before messaging. Facebook is pretty reliable, no? That's the first major outage I've heard of. I see no reason to change. It's not like my parents would love to learn how to use Signal anyway.
This has been a quest of ours for the better part of a year. The only remaining holdouts are my aunt and a couple of stubborn friends. It really shouldn't be this hard. In the 90s we changed messaging platforms like socks
Well I bought 1kg of fresh sardines and only my father knows how to make them the way I like. He only uses Messenger. I am now eating subpar sardines. Thank you, Facebook.
I'm in a similar boat. I got some good news today and I wanted to share it with my mother but I can't because we're only able to communicate via WhatsApp. Shucks.
I'm sorry but you can't reach your own mother besides whatsapp? She doesn't have a phone?
I'm not the person you were asking but just as an example of how this might happen - my mum does have a phone, but she also has terrible mobile reception in her house. She has good wifi though, so Whatsapp is the most reliable way I can reach her. Calls often fail, texts sometimes take a day or more, especially since covid because she's been going out less (50m up the road from her house she has five bars - there's some geography going on...)
At risk of being one of those guys who throws excess tech at anything vaguely looking like a problem: she might benefit from setting up WiFi calling & SMS. Most UK networks support it, as do a significant chunk of phones (that list isn't exhaustive, but it's a great guide to roughly where to look), so it'd be a good emergency backup for situations exactly like yesterday.
[Edit] cc @kwyjibo, in case it's relevant/helpful to you!
Unfortunately my entire family is on GiffGaff, no wifi calling support from them yet, but I'm sure they'll get there eventually. However, it's something I didn't know about and am glad I now do so thanks for posting.
My Mum's cheap Samsung phone is part of the problem, because my Dad's Pixel 3a is totally fine. In an emergency I can always call him. But he's really awkward to talk to on the phone for some reason.
I am aware of Wi-Fi calling, I use it myself. Unfortunately, my mother's phone doesn't support it. I honestly didn't bother with a backup method because her farm is a 20 minutes drive from where I live and there are at least 5 other people on the farm at any given time besides her, so if there is an emergency, I will reach her.
I do appreciate the advice though. I didn't go out of my way to provide her with a backup method, but we'll probably get her a new phone next year so we might as well make sure it's something that supports Wi-Fi calling.
Well shit...I'm never going to get used to the thought of no signal so many places in the states.
My mum lives in the Scottish highlands. Which, perhaps surprisingly, tend to have pretty good signal in my experience. It really is just one end of the road where she lives that is a dead zone. And then only really for her phone, my dad's phone is fine.
Well, she does if she has WhatsApp. I can't speak for the other user, but I live halfway around the world from my parents. Unless it's life or death, I'm not going to be making regular phone calls to them, I'd just wait for WhatsApp or whatever we use to come back up (we actually use Telegram more).
She lives on a farm in a remote area, so there's no cell signal. We have to rely on WhatsApp exclusively to communicate with each other.
There you have it. It's just so weird that there are so many places in the us that doesn't have signal.
I'm neither American nor live in the US. 😊
Whatever happened, Facebook appears to be back up.
I would anticipate it being some time until everything is working normally. It's not possible to know for certain, but it's likely from what we know about how complete the failures were that the bad route update that excised Facebook from the Internet also completely disrupted their internal networking. Every system which needed to talk over a routed network to another system most likely couldn't. It's rare for services to recover gracefully from that sort of thing, so they will be restarting services and ironing out glitches for some time still.
I am really surprised reading comments of people who rather wait any time it takes for Facebook and WhatsApp go online again than use regular phone calls or SMS. This proved how deep is Facebook and its ecosystem integrated on people lives even knowing how shit Facebook it is.
This is a hard battle not even Telegram can win it and Signal is a lost case.
sms and phone calls don't work for everyone, especially those with international family/friends. And phone calls don't allow video. Not sure why you've given up on Telegram and Signal. I'm slowly converting all my network to signal in the hope of one day deleting whatsapp
I ripped off the bandaid and deleted Whatsapp, leaving anyone who I don’t have on Signal or SMS behind.
A few have told me I didn’t answer them on WhatsApp but it’s been relatively painless.
Brave man! I did something less extreme and changed my profile pic to: Look for me on Signal. Gonna leave this up for a month or 20 then delete it.
I used to live abroad in other continent and phone calls, SMS and email were fine back then. When i do business i still give my phone number and people still call me by phone call, i would like to know what makes people stop not using phone calls on this rare circumstances. I do use Telegram and WhatsApp btw.
International phone calls and sms cost extra and don't support video
I don't really understand this framing, honestly. Obviously the network effect is an important thing, but I use both Telegram and Signal with non-techy family and friends on the regular. It works for us. Why does it have to "win" in a global ecosystem? Why can't we just have multiple products, preferably with some level of interoperability?
I need to rephrase a little better, but win i meant a battle versus Facebook ecosystem. Multiple products as messaging apps has always been the case since ICQ.
I've read a few articles saying that this was caused by a DNS failure. I kind of understand that, but what I don't understand is why this is happening. What are the possible reasons for this?
Ars have a being-updated article too. Via there, a since-deleted reddit comment from an alleged FB engineer:
I haven't messaged my friend who works there because either he's in full panic mode and firefighting, or he doesn't know anything which isn't already public. Or possibly both.
Depending on what team they're on its very possible they're just chilling because there's nothing they can do.
I'm not going to lie, it's weirdly entertaining being in a sev1 or sev0 incident when it has nothing to do with your team. It's that fly on the wall feeling. Shits going bad, buuut nothing you can do about it.
Agreed, the vast majority of FB are just watching now. Lots of them probably not even that, unless they're working from home. There's a small handful of very sweaty sysadmins somewhere though. I feel for them. Although it is still fun to watch. I've been on both sides and I definitely prefer the "feet up, can't work even if I wanted to, not my problem" side!
Still too early to tell. Kreb's article is the best you can get for what we know right now.
https://krebsonsecurity.com/2021/10/what-happened-to-facebook-instagram-whatsapp/
Cloudflare's article is a bit more technical, if that's what anyone wants.
The super brief summary is that they pulled their BGP routes (this should be sounding very familiar to outage connoisseurs…), which removed their DNS servers from the Internet, therefore no DNS (but the missing DNS was a symptom, not the root cause).
Someone messed up a patch, and then QA messed up their check, and now Facebook is all sorts of messed up.
We will see. It's a fascinatingly bad failure, in a "you can only see the tip of the iceberg" sort of way.
I've seen people wondering if it's an attack and though I don't personally think it would be, it's plausible. Everything is at this point. Hell maybe it's @kfwyre trying to convert more people to screenless day.
I love this headcanon where I'm a ridiculously capable white hat hacker who is able to bring down the biggest tech company in the world in the interest of helping the public at large disrupt the presence of toxic tech in their lives. :)
The reality is much more down to earth. Just today I had to spend >10 minutes in a meeting walking some of my coworkers through how to attach a PDF to an online assignment -- click by excruciating click.
The patience to walk tech illiterate people through anything is as impressive as hacking, imo! It’s so damn frustrating.
It's an astonishingly bad failure. Especially if the stories about employees not being able to get into FB buildings is true. But even if they're not, someone still managed to take down three of the world's biggest internet services and FB's internal comms with, essentially, a single command. They have failsafes in place for so many things at so many levels on their network but this level of fuckupery is amazing.
I feel bad for their admins. Especially whoever pressed enter and then watched it all collapse. I can only imagine the sinking feeling as the entire monitor board started to light up, more and more and more and then ran out of lights and probably just caught fire.
I've had bad sysadmin days. I've had a few very bad sysadmin days that I still occasionally get mild panic attacks about - but nothing like this.
edit: holy crap look at this chart from Cloudflare, the whole lot fell down in two minutes flat. Madness.
Curious to know if anyone with networking experience knows what needs to be done to fix this?
At this point, I'm assuming that any simple "roll it back" fix is not feasible and people are sufficient locked out of fixing this remotely. Is someone basically having to fix this by physically patching each server/router?
Without knowing what’s gone wrong no one could give a reasonable explanation of what’s needed to fix this. A website being down tells you about as much as a car failing to start.
https://youtu.be/KLqH8p1QVpY
Jeff Geerling just put this quick explainer out on what we know so far. The going theory is that the BGP was affected, and it's affecting everything from remote tools to door badges. Couple that with less people on site who aren't as specialized, and you have a big old disaster.
This is quite possibly the most predictable, lowest effort comment section I've ever seen on a Tildes submission. And the fact that so many users felt the comments celebrating this were worthy of a vote, instead of being noise labeled, is incredibly disappointing.
Edit: Well, that turned around quick. Goes to show the power of the noise label, I suppose... when people actually bother to use it anyways.
Personally I'd rather you reply to me directly to express your displeasure than post this kind of comment, but that's just me. Meta commentary like this is just as "noisy" and non-contributing as what I posted in my opinion, and I don't think that's controversial to say.
I'll also happily point out that I was the first to point out the suspected technical reasons for this downtime in a since noise-nuked subthread, which is more than you've contributed.
I appreciate your sentiment here but I don't want things to get heated. "which is more than you've contributed" isn't necessary to get your point across.
I'm still trying to work out what was wrong with your comment exactly. Is voicing an opinion or having a dig at a company frowned upon around here? The majority of the comments seem to be about the technical side of it anyway, but I cannot personally blame anyone for celebrating the witch being (temporarily) dead, so to speak.
One liners, jokes, quips, "I agree", and others that would reasonably be considered "low effort" are generally frowned upon here. Other users can label such comments as "Joke" or "Noise", which under some criteria will auto-collapse that comment (other labels are "Offtopic", "Malice", and "Exemplary"). There's no need to write an essay, but the goal is to have all comments meaningfully contribute to the discussion in some way.
I tagged them all as joke, but felt similarly. There's no resistance to the slow slippery slope into Reddit 2.0, but there's a lot of reasons why it's been headed that way for awhile.
Eh, I feel like we are still a looooong way from even being 2010 Reddit. Having a few low effort one-liners every once in a while isn't so bad as long as there is still thoughtful discussion going on in most places.
While this is true, the problem is that this kind of stuff tends to grow on it's own if noone steps in.
I've been told that Reddit had always been accepting of humor (at least when comments became a thing). It's not like it was HackerNews in its early days.
I understand, though, that Tildes had a different starting off point. I will say it's been interesting to see the progression of this place. Not that I was here in it's early days as I had no idea about this place's existence until a few months ago. But reading through older posts I've noticed that the style of the comments have been changing. I now see a lot more abbreviations being used, specifically "lol." And emoticons are used now as well. That does feel like it clashes with the intended goals of this forum, even aesthetically it feels odd to see someone type out ":P." So I can see the frustration that older users have with the small scale Eternal September that's been going on.
To be absolutely clear I believe that relaxing some of the rules around humor and formality has been good for this place. The frustration is not about that, but the slow erosion of shared values on this website which has been repeatedly chipped away by bad actors who regularly flirt with what is considered acceptable. There is only so much resistance that can be offered before it takes too high of an emotional toll on the groups attempting to protect civility and without more active moderation and a set of tools and stronger and regularly enforced guidelines this erosion happens to any platform of scale. Assholes will always win out on nice people because nice people would rather go somewhere else when they aren't treated with respect whereas assholes often do not care if they are disrespected as they don't value others opinions as much as their own.
Yeah, very reminiscent of Reddit. A lot of low effort quips in the vein of “Facebook bad”.
Should we have a policy of removing submissions that people only have low quality responses to?
I don't believe that's necessary. A submission that inspires a bunch of low effort comments can still be newsworthy and worth sharing, as was the case here. I think people might just need to occasionally be reminded that the noise label exists in order to counteract that trend, as was also the case here too.
And stay down!
Here's to hoping that this is a recurring problem that remains impossible to handle for them.
It looks like a DNS issue. While embarrassing, it is relatively a simple problem
It's worse than DNS. The IP address range that points to Facebook's DNS servers no longer does so. It's not something I've personally ever worked with, but this is a BGP problem, apparently.