53 votes

Amazon Web Services outage impacts

26 comments

  1. [16]
    TheMediumJon
    Link
    So apparently the US-East AWS servers are having issues.... In the spirit of discussion? How's this impacting you, if at all? Personally, slack as well as a bunch of other company services are...

    So apparently the US-East AWS servers are having issues....

    In the spirit of discussion? How's this impacting you, if at all?

    Personally, slack as well as a bunch of other company services are barely functional or not at all. Great fun.

    17 votes
    1. [3]
      Ozzy
      Link Parent
      Can’t access Signal, can’t access my bank account, can’t access a couple more stuff. I’m in Europe, damn you, AWS.

      Can’t access Signal, can’t access my bank account, can’t access a couple more stuff. I’m in Europe, damn you, AWS.

      14 votes
      1. Greg
        Link Parent
        Ohhhh, that's why Signal says I'm offline! Glad I don't have to go digging into firewall settings or anything.

        Ohhhh, that's why Signal says I'm offline! Glad I don't have to go digging into firewall settings or anything.

        10 votes
      2. TheMediumJon
        Link Parent
        Oh man, I've not tried anything personal at all yet, actually.

        Oh man, I've not tried anything personal at all yet, actually.

        5 votes
    2. redwall_hp
      Link Parent
      I've barely slept. I got paged over and over all night from the metric monitors on my team's apps freaking out from zero traffic in our east region. Our product is chugging along, though, because...

      I've barely slept. I got paged over and over all night from the metric monitors on my team's apps freaking out from zero traffic in our east region. Our product is chugging along, though, because we did a region failover.

      Basically a cycle of trying to nap for a few minutes, acknowledging a page, confirming it's the same thing, playing mute whack-a-mole on the monitor, resolving the page, catching up on what other teams are doing for the mitigation, repeat.

      And Slack is slow.

      12 votes
    3. [3]
      Macha
      Link Parent
      No docker, no npm, slack is a mess. IAM is causing problems in work even though we’re in eu regions

      No docker, no npm, slack is a mess.

      IAM is causing problems in work even though we’re in eu regions

      11 votes
      1. [2]
        Crestwave
        Link Parent
        It's quite illuminating seeing how much infrastructure all over the world was affected from the outage of a single us-east-1 service. Hopefully some companies will consider lessening their...

        It's quite illuminating seeing how much infrastructure all over the world was affected from the outage of a single us-east-1 service. Hopefully some companies will consider lessening their dependency on a single region.

        17 votes
    4. shrike
      Link Parent
      Docker is down, Slack is slow. I'm on a long lunch break.

      Docker is down, Slack is slow.

      I'm on a long lunch break.

      7 votes
    5. blackforest
      Link Parent
      I can’t send or receive pictures on Snapchat since this morning. Messages on group chats work, but every sender is now marked as “Unknown Snapchatter” rather than their username, so I can’t follow...

      I can’t send or receive pictures on Snapchat since this morning. Messages on group chats work, but every sender is now marked as “Unknown Snapchatter” rather than their username, so I can’t follow the conversation.

      6 votes
    6. [3]
      Fal
      Link Parent
      Canvas is down I HAVE HOMEWORK TO SUBMIT AAAAAAAAAA

      Canvas is down I HAVE HOMEWORK TO SUBMIT AAAAAAAAAA

      6 votes
      1. [2]
        CannibalisticApple
        Link Parent
        I feel like that has to be forgiveable by the instructors if it's not back up before the deadline.

        I feel like that has to be forgiveable by the instructors if it's not back up before the deadline.

        9 votes
    7. erithaea
      Link Parent
      Atlassian auth services are/were down, so I couldn't access Jira at work for the entire morning. It's working again now (or at least it worked for long enough to allow me to log in) but since...

      Atlassian auth services are/were down, so I couldn't access Jira at work for the entire morning. It's working again now (or at least it worked for long enough to allow me to log in) but since we're currently in a QA period where we have to create and review a lot of tickets, it was a pretty big showstopper.

      4 votes
    8. ras
      Link Parent
      really wild how long this has gone on. i just got back from taking my son to the ENT and their payment processing was down due to the outage. getting close to 11 hours since the first reports.

      really wild how long this has gone on. i just got back from taking my son to the ENT and their payment processing was down due to the outage. getting close to 11 hours since the first reports.

      1 vote
    9. lou
      Link Parent
      Mercado Livre, Brazil's Ebay, cannot show when my delivery is arriving.

      Mercado Livre, Brazil's Ebay, cannot show when my delivery is arriving.

  2. trim
    Link
    I work in tech deploying software to customers through AWS. It's been a morning. Fortunately not too much impact on eu-west but global services and our access to console, and monitoring were...

    I work in tech deploying software to customers through AWS. It's been a morning. Fortunately not too much impact on eu-west but global services and our access to console, and monitoring were affected. Kind of like being in the dark whilst fielding questions we can't answer about metrics we can't see.

    15 votes
  3. hamstergeddon
    (edited )
    Link
    It's times like these that I'm glad I never segued into devops. But then I'd have a bit more money to cry into about it if I had. Edit- as it stands I can't access my Jira board which is going to...

    It's times like these that I'm glad I never segued into devops. But then I'd have a bit more money to cry into about it if I had.

    Edit- as it stands I can't access my Jira board which is going to make for an interesting stand up meeting in about 15 minutes :)

    5 votes
  4. redwall_hp
    Link
    Fun take from The Register: Today is when the Amazon brain drain finally sent AWS down the spout Good to call out the three solid years of layoffs and forced attrition through return-to-office...

    Fun take from The Register: Today is when the Amazon brain drain finally sent AWS down the spout

    Good to call out the three solid years of layoffs and forced attrition through return-to-office policies at Amazon...

    5 votes
  5. [3]
    Eji1700
    Link
    As an shop that uses azure instead of AWS, bullet dodged internally BUT naturally our 3rd party vendors use AWS, so still a mess from upstream data.

    As an shop that uses azure instead of AWS, bullet dodged internally BUT naturally our 3rd party vendors use AWS, so still a mess from upstream data.

    2 votes
    1. [2]
      tanglisha
      Link Parent
      Here’s an article from 2019 by someone who tried to completely block Amazon from their life.
      5 votes
      1. CannibalisticApple
        Link Parent
        Wow. That was a fascinating read, and probably worthy of its own individual topic. It highlights Amazon really is too powerful given how many services it can impact.

        Wow. That was a fascinating read, and probably worthy of its own individual topic. It highlights Amazon really is too powerful given how many services it can impact.

        3 votes
  6. [4]
    TheFireTheft
    Link
    I couldn't find anything in the article about the root cause, but let me guess: Config related? Did somebody mess up a YAML file?

    I couldn't find anything in the article about the root cause, but let me guess:

    Config related?

    Did somebody mess up a YAML file?

    2 votes
    1. [4]
      Comment deleted by author
      Link Parent
      1. [3]
        first-must-burn
        Link Parent
        It's always DNS.

        It's always DNS.

        11 votes
        1. [2]
          trim
          Link Parent
          BGP enters stage left, glancing furtively.

          BGP enters stage left, glancing furtively.

          4 votes
          1. first-must-burn
            Link Parent
            DNS stands on BGP's shoulders in a trench coat, and they try to pass as an application level error.

            DNS stands on BGP's shoulders in a trench coat, and they try to pass as an application level error.

            3 votes