47 votes

Facebook, Messenger, Instagram and WhatsApp are all down

63 comments

  1. [2]
    Adys
    Link
    Signal and Telegram are both extremely slow, most likely due to knock-on effects.

    Signal and Telegram are both extremely slow, most likely due to knock-on effects.

    17 votes
    1. petrichor
      Link Parent
      Discord doesn't appear to be affected. I did read somewhere they were dealing with the extra load by disabling non-essential server load (like typing notifications).

      Discord doesn't appear to be affected. I did read somewhere they were dealing with the extra load by disabling non-essential server load (like typing notifications).

      6 votes
  2. Kuromantis
    (edited )
    Link
    A relevant article on how other sites aren't taking the current extra load of people well: Twitter, Is It Down, and tons of other sites are struggling due to the Facebook outage This is many...

    A relevant article on how other sites aren't taking the current extra load of people well: Twitter, Is It Down, and tons of other sites are struggling due to the Facebook outage

    This is many things that I don't know, but I find it most glaringly a decent demonstration of why centralized social media websites are bad, given even the Oculus store is down.

    13 votes
  3. [7]
    NaraVara
    Link
    This may be the time to finally kill my family's WhatsApp group chat and move them to Signal. But alas. . . I have no way to reach them.

    This may be the time to finally kill my family's WhatsApp group chat and move them to Signal.

    But alas. . . I have no way to reach them.

    10 votes
    1. [5]
      skybrian
      Link Parent
      You don't have their phone numbers? It might be good to fix that.

      You don't have their phone numbers? It might be good to fix that.

      7 votes
      1. moriarty
        Link Parent
        Sure I do. But that's the nature of messaging apps. Nobody wants to move to different network that doesn't have all their friends

        Sure I do. But that's the nature of messaging apps. Nobody wants to move to different network that doesn't have all their friends

        4 votes
      2. [4]
        Comment deleted by author
        Link Parent
        1. [2]
          skybrian
          Link Parent
          There are also text messages, and email (if you have everyone’s email).

          There are also text messages, and email (if you have everyone’s email).

          8 votes
          1. guts
            Link Parent
            Man now the sudden people forgot about phone calls and SMS.

            Man now the sudden people forgot about phone calls and SMS.

            7 votes
        2. joplin
          Link Parent
          I mean, if you're not willing to put in a little effort to help them along, why should they put in the effort to change to a different messenger app?

          I mean, if you're not willing to put in a little effort to help them along, why should they put in the effort to change to a different messenger app?

          2 votes
    2. moriarty
      Link Parent
      This has been a quest of ours for the better part of a year. The only remaining holdouts are my aunt and a couple of stubborn friends. It really shouldn't be this hard. In the 90s we changed...

      This has been a quest of ours for the better part of a year. The only remaining holdouts are my aunt and a couple of stubborn friends. It really shouldn't be this hard. In the 90s we changed messaging platforms like socks

      5 votes
  4. [13]
    Comment deleted by author
    Link
    1. [12]
      kwyjibo
      Link Parent
      I'm in a similar boat. I got some good news today and I wanted to share it with my mother but I can't because we're only able to communicate via WhatsApp. Shucks.

      I'm in a similar boat. I got some good news today and I wanted to share it with my mother but I can't because we're only able to communicate via WhatsApp. Shucks.

      4 votes
      1. [11]
        elcuello
        Link Parent
        I'm sorry but you can't reach your own mother besides whatsapp? She doesn't have a phone?

        I'm sorry but you can't reach your own mother besides whatsapp? She doesn't have a phone?

        2 votes
        1. [6]
          mat
          Link Parent
          I'm not the person you were asking but just as an example of how this might happen - my mum does have a phone, but she also has terrible mobile reception in her house. She has good wifi though, so...

          I'm not the person you were asking but just as an example of how this might happen - my mum does have a phone, but she also has terrible mobile reception in her house. She has good wifi though, so Whatsapp is the most reliable way I can reach her. Calls often fail, texts sometimes take a day or more, especially since covid because she's been going out less (50m up the road from her house she has five bars - there's some geography going on...)

          8 votes
          1. [3]
            Greg
            (edited )
            Link Parent
            At risk of being one of those guys who throws excess tech at anything vaguely looking like a problem: she might benefit from setting up WiFi calling & SMS. Most UK networks support it, as do a...

            At risk of being one of those guys who throws excess tech at anything vaguely looking like a problem: she might benefit from setting up WiFi calling & SMS. Most UK networks support it, as do a significant chunk of phones (that list isn't exhaustive, but it's a great guide to roughly where to look), so it'd be a good emergency backup for situations exactly like yesterday.

            [Edit] cc @kwyjibo, in case it's relevant/helpful to you!

            7 votes
            1. mat
              Link Parent
              Unfortunately my entire family is on GiffGaff, no wifi calling support from them yet, but I'm sure they'll get there eventually. However, it's something I didn't know about and am glad I now do so...

              Unfortunately my entire family is on GiffGaff, no wifi calling support from them yet, but I'm sure they'll get there eventually. However, it's something I didn't know about and am glad I now do so thanks for posting.

              My Mum's cheap Samsung phone is part of the problem, because my Dad's Pixel 3a is totally fine. In an emergency I can always call him. But he's really awkward to talk to on the phone for some reason.

              4 votes
            2. kwyjibo
              Link Parent
              I am aware of Wi-Fi calling, I use it myself. Unfortunately, my mother's phone doesn't support it. I honestly didn't bother with a backup method because her farm is a 20 minutes drive from where I...

              I am aware of Wi-Fi calling, I use it myself. Unfortunately, my mother's phone doesn't support it. I honestly didn't bother with a backup method because her farm is a 20 minutes drive from where I live and there are at least 5 other people on the farm at any given time besides her, so if there is an emergency, I will reach her.

              I do appreciate the advice though. I didn't go out of my way to provide her with a backup method, but we'll probably get her a new phone next year so we might as well make sure it's something that supports Wi-Fi calling.

              3 votes
          2. [2]
            elcuello
            Link Parent
            Well shit...I'm never going to get used to the thought of no signal so many places in the states.

            Well shit...I'm never going to get used to the thought of no signal so many places in the states.

            1. mat
              Link Parent
              My mum lives in the Scottish highlands. Which, perhaps surprisingly, tend to have pretty good signal in my experience. It really is just one end of the road where she lives that is a dead zone....

              My mum lives in the Scottish highlands. Which, perhaps surprisingly, tend to have pretty good signal in my experience. It really is just one end of the road where she lives that is a dead zone. And then only really for her phone, my dad's phone is fine.

              3 votes
        2. aditya
          Link Parent
          Well, she does if she has WhatsApp. I can't speak for the other user, but I live halfway around the world from my parents. Unless it's life or death, I'm not going to be making regular phone calls...

          Well, she does if she has WhatsApp. I can't speak for the other user, but I live halfway around the world from my parents. Unless it's life or death, I'm not going to be making regular phone calls to them, I'd just wait for WhatsApp or whatever we use to come back up (we actually use Telegram more).

          7 votes
        3. [3]
          kwyjibo
          Link Parent
          She lives on a farm in a remote area, so there's no cell signal. We have to rely on WhatsApp exclusively to communicate with each other.

          She lives on a farm in a remote area, so there's no cell signal. We have to rely on WhatsApp exclusively to communicate with each other.

          7 votes
          1. [2]
            elcuello
            Link Parent
            There you have it. It's just so weird that there are so many places in the us that doesn't have signal.

            There you have it. It's just so weird that there are so many places in the us that doesn't have signal.

            1. kwyjibo
              Link Parent
              I'm neither American nor live in the US. 😊

              I'm neither American nor live in the US. 😊

              4 votes
  5. [2]
    moocow1452
    Link
    Whatever happened, Facebook appears to be back up.

    Whatever happened, Facebook appears to be back up.

    5 votes
    1. [2]
      Comment deleted by author
      Link Parent
      1. whbboyd
        Link Parent
        I would anticipate it being some time until everything is working normally. It's not possible to know for certain, but it's likely from what we know about how complete the failures were that the...

        I would anticipate it being some time until everything is working normally. It's not possible to know for certain, but it's likely from what we know about how complete the failures were that the bad route update that excised Facebook from the Internet also completely disrupted their internal networking. Every system which needed to talk over a routed network to another system most likely couldn't. It's rare for services to recover gracefully from that sort of thing, so they will be restarting services and ironing out glitches for some time still.

        8 votes
  6. [8]
    guts
    Link
    I am really surprised reading comments of people who rather wait any time it takes for Facebook and WhatsApp go online again than use regular phone calls or SMS. This proved how deep is Facebook...

    I am really surprised reading comments of people who rather wait any time it takes for Facebook and WhatsApp go online again than use regular phone calls or SMS. This proved how deep is Facebook and its ecosystem integrated on people lives even knowing how shit Facebook it is.

    This is a hard battle not even Telegram can win it and Signal is a lost case.

    5 votes
    1. [5]
      moriarty
      Link Parent
      sms and phone calls don't work for everyone, especially those with international family/friends. And phone calls don't allow video. Not sure why you've given up on Telegram and Signal. I'm slowly...

      sms and phone calls don't work for everyone, especially those with international family/friends. And phone calls don't allow video. Not sure why you've given up on Telegram and Signal. I'm slowly converting all my network to signal in the hope of one day deleting whatsapp

      7 votes
      1. [2]
        GoingMerry
        Link Parent
        I ripped off the bandaid and deleted Whatsapp, leaving anyone who I don’t have on Signal or SMS behind. A few have told me I didn’t answer them on WhatsApp but it’s been relatively painless.

        I ripped off the bandaid and deleted Whatsapp, leaving anyone who I don’t have on Signal or SMS behind.

        A few have told me I didn’t answer them on WhatsApp but it’s been relatively painless.

        2 votes
        1. moriarty
          Link Parent
          Brave man! I did something less extreme and changed my profile pic to: Look for me on Signal. Gonna leave this up for a month or 20 then delete it.

          Brave man! I did something less extreme and changed my profile pic to: Look for me on Signal. Gonna leave this up for a month or 20 then delete it.

      2. [2]
        guts
        Link Parent
        I used to live abroad in other continent and phone calls, SMS and email were fine back then. When i do business i still give my phone number and people still call me by phone call, i would like to...

        I used to live abroad in other continent and phone calls, SMS and email were fine back then. When i do business i still give my phone number and people still call me by phone call, i would like to know what makes people stop not using phone calls on this rare circumstances. I do use Telegram and WhatsApp btw.

        1. moriarty
          Link Parent
          International phone calls and sms cost extra and don't support video

          International phone calls and sms cost extra and don't support video

          5 votes
    2. [2]
      mtset
      Link Parent
      I don't really understand this framing, honestly. Obviously the network effect is an important thing, but I use both Telegram and Signal with non-techy family and friends on the regular. It works...

      I don't really understand this framing, honestly. Obviously the network effect is an important thing, but I use both Telegram and Signal with non-techy family and friends on the regular. It works for us. Why does it have to "win" in a global ecosystem? Why can't we just have multiple products, preferably with some level of interoperability?

      4 votes
      1. guts
        Link Parent
        I need to rephrase a little better, but win i meant a battle versus Facebook ecosystem. Multiple products as messaging apps has always been the case since ICQ.

        I need to rephrase a little better, but win i meant a battle versus Facebook ecosystem. Multiple products as messaging apps has always been the case since ICQ.

  7. [11]
    cloud_loud
    (edited )
    Link
    I've read a few articles saying that this was caused by a DNS failure. I kind of understand that, but what I don't understand is why this is happening. What are the possible reasons for this?

    I've read a few articles saying that this was caused by a DNS failure. I kind of understand that, but what I don't understand is why this is happening. What are the possible reasons for this?

    3 votes
    1. [3]
      mat
      Link Parent
      Ars have a being-updated article too. Via there, a since-deleted reddit comment from an alleged FB engineer: I haven't messaged my friend who works there because either he's in full panic mode and...

      Ars have a being-updated article too. Via there, a since-deleted reddit comment from an alleged FB engineer:

      the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

      I haven't messaged my friend who works there because either he's in full panic mode and firefighting, or he doesn't know anything which isn't already public. Or possibly both.

      10 votes
      1. [2]
        stu2b50
        Link Parent
        Depending on what team they're on its very possible they're just chilling because there's nothing they can do. I'm not going to lie, it's weirdly entertaining being in a sev1 or sev0 incident when...

        Depending on what team they're on its very possible they're just chilling because there's nothing they can do.

        I'm not going to lie, it's weirdly entertaining being in a sev1 or sev0 incident when it has nothing to do with your team. It's that fly on the wall feeling. Shits going bad, buuut nothing you can do about it.

        13 votes
        1. mat
          Link Parent
          Agreed, the vast majority of FB are just watching now. Lots of them probably not even that, unless they're working from home. There's a small handful of very sweaty sysadmins somewhere though. I...

          Agreed, the vast majority of FB are just watching now. Lots of them probably not even that, unless they're working from home. There's a small handful of very sweaty sysadmins somewhere though. I feel for them. Although it is still fun to watch. I've been on both sides and I definitely prefer the "feet up, can't work even if I wanted to, not my problem" side!

          9 votes
    2. [2]
      Adys
      Link Parent
      Still too early to tell. Kreb's article is the best you can get for what we know right now. https://krebsonsecurity.com/2021/10/what-happened-to-facebook-instagram-whatsapp/

      Still too early to tell. Kreb's article is the best you can get for what we know right now.

      https://krebsonsecurity.com/2021/10/what-happened-to-facebook-instagram-whatsapp/

      7 votes
      1. whbboyd
        Link Parent
        Cloudflare's article is a bit more technical, if that's what anyone wants. The super brief summary is that they pulled their BGP routes (this should be sounding very familiar to outage...

        Cloudflare's article is a bit more technical, if that's what anyone wants.

        The super brief summary is that they pulled their BGP routes (this should be sounding very familiar to outage connoisseurs…), which removed their DNS servers from the Internet, therefore no DNS (but the missing DNS was a symptom, not the root cause).

        7 votes
    3. [5]
      moocow1452
      Link Parent
      Someone messed up a patch, and then QA messed up their check, and now Facebook is all sorts of messed up.

      Someone messed up a patch, and then QA messed up their check, and now Facebook is all sorts of messed up.

      2 votes
      1. [4]
        Adys
        Link Parent
        We will see. It's a fascinatingly bad failure, in a "you can only see the tip of the iceberg" sort of way. I've seen people wondering if it's an attack and though I don't personally think it would...

        We will see. It's a fascinatingly bad failure, in a "you can only see the tip of the iceberg" sort of way.

        I've seen people wondering if it's an attack and though I don't personally think it would be, it's plausible. Everything is at this point. Hell maybe it's @kfwyre trying to convert more people to screenless day.

        9 votes
        1. [2]
          kfwyre
          (edited )
          Link Parent
          I love this headcanon where I'm a ridiculously capable white hat hacker who is able to bring down the biggest tech company in the world in the interest of helping the public at large disrupt the...

          I love this headcanon where I'm a ridiculously capable white hat hacker who is able to bring down the biggest tech company in the world in the interest of helping the public at large disrupt the presence of toxic tech in their lives. :)

          The reality is much more down to earth. Just today I had to spend >10 minutes in a meeting walking some of my coworkers through how to attach a PDF to an online assignment -- click by excruciating click.

          18 votes
          1. hamstergeddon
            Link Parent
            The patience to walk tech illiterate people through anything is as impressive as hacking, imo! It’s so damn frustrating.

            The patience to walk tech illiterate people through anything is as impressive as hacking, imo! It’s so damn frustrating.

            8 votes
        2. mat
          (edited )
          Link Parent
          It's an astonishingly bad failure. Especially if the stories about employees not being able to get into FB buildings is true. But even if they're not, someone still managed to take down three of...

          It's an astonishingly bad failure. Especially if the stories about employees not being able to get into FB buildings is true. But even if they're not, someone still managed to take down three of the world's biggest internet services and FB's internal comms with, essentially, a single command. They have failsafes in place for so many things at so many levels on their network but this level of fuckupery is amazing.

          I feel bad for their admins. Especially whoever pressed enter and then watched it all collapse. I can only imagine the sinking feeling as the entire monitor board started to light up, more and more and more and then ran out of lights and probably just caught fire.

          I've had bad sysadmin days. I've had a few very bad sysadmin days that I still occasionally get mild panic attacks about - but nothing like this.

          edit: holy crap look at this chart from Cloudflare, the whole lot fell down in two minutes flat. Madness.

          11 votes
  8. [3]
    symmetry
    Link
    Curious to know if anyone with networking experience knows what needs to be done to fix this? At this point, I'm assuming that any simple "roll it back" fix is not feasible and people are...

    Curious to know if anyone with networking experience knows what needs to be done to fix this?

    At this point, I'm assuming that any simple "roll it back" fix is not feasible and people are sufficient locked out of fixing this remotely. Is someone basically having to fix this by physically patching each server/router?

    2 votes
    1. teaearlgraycold
      Link Parent
      Without knowing what’s gone wrong no one could give a reasonable explanation of what’s needed to fix this. A website being down tells you about as much as a car failing to start.

      Without knowing what’s gone wrong no one could give a reasonable explanation of what’s needed to fix this. A website being down tells you about as much as a car failing to start.

      9 votes
    2. moocow1452
      Link Parent
      https://youtu.be/KLqH8p1QVpY Jeff Geerling just put this quick explainer out on what we know so far. The going theory is that the BGP was affected, and it's affecting everything from remote tools...

      https://youtu.be/KLqH8p1QVpY

      Jeff Geerling just put this quick explainer out on what we know so far. The going theory is that the BGP was affected, and it's affecting everything from remote tools to door badges. Couple that with less people on site who aren't as specialized, and you have a big old disaster.

      5 votes
  9. [13]
    cfabbro
    (edited )
    Link
    This is quite possibly the most predictable, lowest effort comment section I've ever seen on a Tildes submission. And the fact that so many users felt the comments celebrating this were worthy of...

    This is quite possibly the most predictable, lowest effort comment section I've ever seen on a Tildes submission. And the fact that so many users felt the comments celebrating this were worthy of a vote, instead of being noise labeled, is incredibly disappointing.

    Edit: Well, that turned around quick. Goes to show the power of the noise label, I suppose... when people actually bother to use it anyways.

    17 votes
    1. [4]
      jcdl
      Link Parent
      Personally I'd rather you reply to me directly to express your displeasure than post this kind of comment, but that's just me. Meta commentary like this is just as "noisy" and non-contributing as...

      Personally I'd rather you reply to me directly to express your displeasure than post this kind of comment, but that's just me. Meta commentary like this is just as "noisy" and non-contributing as what I posted in my opinion, and I don't think that's controversial to say.

      I'll also happily point out that I was the first to point out the suspected technical reasons for this downtime in a since noise-nuked subthread, which is more than you've contributed.

      13 votes
      1. teaearlgraycold
        Link Parent
        I appreciate your sentiment here but I don't want things to get heated. "which is more than you've contributed" isn't necessary to get your point across.

        I appreciate your sentiment here but I don't want things to get heated. "which is more than you've contributed" isn't necessary to get your point across.

        5 votes
      2. [2]
        eldersnake
        Link Parent
        I'm still trying to work out what was wrong with your comment exactly. Is voicing an opinion or having a dig at a company frowned upon around here? The majority of the comments seem to be about...

        I'm still trying to work out what was wrong with your comment exactly. Is voicing an opinion or having a dig at a company frowned upon around here? The majority of the comments seem to be about the technical side of it anyway, but I cannot personally blame anyone for celebrating the witch being (temporarily) dead, so to speak.

        2 votes
        1. DrStone
          Link Parent
          One liners, jokes, quips, "I agree", and others that would reasonably be considered "low effort" are generally frowned upon here. Other users can label such comments as "Joke" or "Noise", which...

          One liners, jokes, quips, "I agree", and others that would reasonably be considered "low effort" are generally frowned upon here. Other users can label such comments as "Joke" or "Noise", which under some criteria will auto-collapse that comment (other labels are "Offtopic", "Malice", and "Exemplary"). There's no need to write an essay, but the goal is to have all comments meaningfully contribute to the discussion in some way.

          8 votes
    2. [5]
      Gaywallet
      Link Parent
      I tagged them all as joke, but felt similarly. There's no resistance to the slow slippery slope into Reddit 2.0, but there's a lot of reasons why it's been headed that way for awhile.

      I tagged them all as joke, but felt similarly. There's no resistance to the slow slippery slope into Reddit 2.0, but there's a lot of reasons why it's been headed that way for awhile.

      6 votes
      1. [4]
        AnthonyB
        Link Parent
        Eh, I feel like we are still a looooong way from even being 2010 Reddit. Having a few low effort one-liners every once in a while isn't so bad as long as there is still thoughtful discussion going...

        Eh, I feel like we are still a looooong way from even being 2010 Reddit. Having a few low effort one-liners every once in a while isn't so bad as long as there is still thoughtful discussion going on in most places.

        12 votes
        1. Kuromantis
          (edited )
          Link Parent
          While this is true, the problem is that this kind of stuff tends to grow on it's own if noone steps in.

          While this is true, the problem is that this kind of stuff tends to grow on it's own if noone steps in.

          3 votes
        2. [2]
          cloud_loud
          Link Parent
          I've been told that Reddit had always been accepting of humor (at least when comments became a thing). It's not like it was HackerNews in its early days. I understand, though, that Tildes had a...

          I've been told that Reddit had always been accepting of humor (at least when comments became a thing). It's not like it was HackerNews in its early days.

          I understand, though, that Tildes had a different starting off point. I will say it's been interesting to see the progression of this place. Not that I was here in it's early days as I had no idea about this place's existence until a few months ago. But reading through older posts I've noticed that the style of the comments have been changing. I now see a lot more abbreviations being used, specifically "lol." And emoticons are used now as well. That does feel like it clashes with the intended goals of this forum, even aesthetically it feels odd to see someone type out ":P." So I can see the frustration that older users have with the small scale Eternal September that's been going on.

          2 votes
          1. Gaywallet
            Link Parent
            To be absolutely clear I believe that relaxing some of the rules around humor and formality has been good for this place. The frustration is not about that, but the slow erosion of shared values...

            To be absolutely clear I believe that relaxing some of the rules around humor and formality has been good for this place. The frustration is not about that, but the slow erosion of shared values on this website which has been repeatedly chipped away by bad actors who regularly flirt with what is considered acceptable. There is only so much resistance that can be offered before it takes too high of an emotional toll on the groups attempting to protect civility and without more active moderation and a set of tools and stronger and regularly enforced guidelines this erosion happens to any platform of scale. Assholes will always win out on nice people because nice people would rather go somewhere else when they aren't treated with respect whereas assholes often do not care if they are disrespected as they don't value others opinions as much as their own.

            10 votes
    3. stu2b50
      Link Parent
      Yeah, very reminiscent of Reddit. A lot of low effort quips in the vein of “Facebook bad”.

      Yeah, very reminiscent of Reddit. A lot of low effort quips in the vein of “Facebook bad”.

      3 votes
    4. [2]
      teaearlgraycold
      Link Parent
      Should we have a policy of removing submissions that people only have low quality responses to?

      Should we have a policy of removing submissions that people only have low quality responses to?

      3 votes
      1. cfabbro
        Link Parent
        I don't believe that's necessary. A submission that inspires a bunch of low effort comments can still be newsworthy and worth sharing, as was the case here. I think people might just need to...

        I don't believe that's necessary. A submission that inspires a bunch of low effort comments can still be newsworthy and worth sharing, as was the case here. I think people might just need to occasionally be reminded that the noise label exists in order to counteract that trend, as was also the case here too.

        7 votes
  10. jcdl
    Link
    And stay down!

    And stay down!

    9 votes
  11. [3]
    drannex
    Link
    Here's to hoping that this is a recurring problem that remains impossible to handle for them.

    Here's to hoping that this is a recurring problem that remains impossible to handle for them.

    8 votes
    1. [2]
      moriarty
      Link Parent
      It looks like a DNS issue. While embarrassing, it is relatively a simple problem

      It looks like a DNS issue. While embarrassing, it is relatively a simple problem

      3 votes
      1. jcdl
        Link Parent
        It's worse than DNS. The IP address range that points to Facebook's DNS servers no longer does so. It's not something I've personally ever worked with, but this is a BGP problem, apparently.

        It's worse than DNS. The IP address range that points to Facebook's DNS servers no longer does so. It's not something I've personally ever worked with, but this is a BGP problem, apparently.

        4 votes