$FB and $AMZN are tanking in kind, bringing the NASDAQ and larger US economy down with it today. Smells like panic selling to me. Big tech is too powerful to get split up by the government like this.
$FB and $AMZN are tanking in kind, bringing the NASDAQ and larger US economy down with it today.
Smells like panic selling to me. Big tech is too powerful to get split up by the government like this.
The S&P is a far better indicator for the "larger US economy" and it's barely down right now. I don't think we should be running around talking about the US economy coming down today. We can...
The S&P is a far better indicator for the "larger US economy" and it's barely down right now. I don't think we should be running around talking about the US economy coming down today.
We can definitely talk about a stagnating US economy in the past 6, and next 18, months.
That outage yesterday smelled a bit fishy. I wonder if someone's in panic mode, cleaning up data, and trying to implement new systems/algorithms in a hurry to get ahead of this probe.
That outage yesterday smelled a bit fishy. I wonder if someone's in panic mode, cleaning up data, and trying to implement new systems/algorithms in a hurry to get ahead of this probe.
That seems like a stretch to me. The outage was disastrous, I can't imagine a single higher up at Google (especially engineers) that would have been comfortable tanking all of their cloud...
That seems like a stretch to me. The outage was disastrous, I can't imagine a single higher up at Google (especially engineers) that would have been comfortable tanking all of their cloud customers and their own services for hours, nor can I imagine how that could possibly be related to this probe.
My only point is that hasty updates make a mistake. Most outages are not existing systems caving in - you can monitor for that and get ahead of it, so it rarely happens. What takes things down is...
My only point is that hasty updates make a mistake. Most outages are not existing systems caving in - you can monitor for that and get ahead of it, so it rarely happens. What takes things down is making changes that you haven't thoroughly prepared for and tested. Even then, mistakes happen, behaviors emerge in the system that can cause it to lose coherence. The more complex, the more likely.
When I see an outage, my immediate reaction is to ask, "What'd you change?" not "What broke?"
Being related to the probe is speculative, but outages being a result of changes is the rule, not the exception. Haste makes that more likely. If Google is being probed, and is feeling like they need to make changes because of it, that could have made them hasty.
On the other hand, someone could have tripped over the wrong cable. That also happens. If youtube's systems start exhibiting new behaviors, I'll be satisfied it was a rocky update. They've been making a lot of changes in the last couple of months, many of the channels I follow are talking about this. They mention videos several years old suddenly getting demonetized, changes in what is advertised in front of their channels, new patterns in the ranking and recommendation systems, plus changes in what's acceptable in chats, all in the last two or three months.
The 'conspiracy' theory I hear is that youtube is gearing up for the election. I wouldn't go that far. Sounds more like google is taking the tech/social responsibility criticism to heart and is starting to make changes to deal with the issues. That's a good thing.
Edit: Wanted to acknowledge that I definitely mis-/over-interpretted your initial statement, so thanks again for explaining further. Thanks for breaking down your thoughts a bit more. I think...
Edit: Wanted to acknowledge that I definitely mis-/over-interpretted your initial statement, so thanks again for explaining further.
Thanks for breaking down your thoughts a bit more. I think there are two reasons that I'm skeptical of even that viewpoint:
This wasn't a YouTube or other specific Google service outage, it was a Google Cloud Networking outage. That affected everything on GCE, and I assume also stuff on Borg, which is why Google services also had outages. That doesn't mean that it's impossible that this is tied to some YouTube updates, but that seems pretty unlikely to me.
The most other recent major cloud-computing outage, AWS's east coast S3 outage from a few years ago, had nothing to do with hasty updates. A new hire used a [poorly written] script incorrectly and turned off orders of magnitude more servers than intended.
I agree that this was probably related to some production change in GCP, but GCP sees so many changes every day. The timing here might be making this seem substantially more interesting than it is.
I'll be reading their incident report with interest, that's for sure. Google doesn't typically make mistakes like this, which is why I'm curious - as you pointed out, it's an expensive mistake and...
I'll be reading their incident report with interest, that's for sure. Google doesn't typically make mistakes like this, which is why I'm curious - as you pointed out, it's an expensive mistake and one to be avoided at all costs.
So far, this seems to have been a routing issue according to the status pages and some comments on HN. That's typically the result of a segment going down, forcing traffic to go elsewhere, and resulting in a traffic jam when the routers get overloaded. It probably wasn't a physical outage (such as a drunk backhoe operator) because they got it back up rather fast - those usually take longer to resolve. I'd like to know what precipitated the network issues.
can we please as a collective not be weirdly conspiratorial like this when there's literally no basis for the conspiratorial thinking? cloud services explode like they did yesterday...
can we please as a collective not be weirdly conspiratorial like this when there's literally no basis for the conspiratorial thinking? cloud services explode like they did yesterday not-infrequently, and it probably has fuck all to do with anything being reported here.
It's hardly a conspiracy. Outages like this correlate to many of youtube's backend algorithm updates and have for years - not all of them, of course. It's just that rejiggering the backend does...
It's hardly a conspiracy. Outages like this correlate to many of youtube's backend algorithm updates and have for years - not all of them, of course. It's just that rejiggering the backend does lead to disruptions in any complex system, that's how it goes in tech. It's hardly unique to google, either.
i would say it is fairly conspiratorial and that there's really no business even going down that road right now, considering that we're not even certain on whether or not a probe is actually...
i would say it is fairly conspiratorial and that there's really no business even going down that road right now, considering that we're not even certain on whether or not a probe is actually happening, to say nothing of the total lack of evidence pointing anywhere in the general direction you're pointing here. google has a billion moving parts and a billion services, and there are a lot of reasons they could run into anti-trust issues that have literally nothing to do with things they'd need to change on their backend, and equally as many reasons that they could randomly fuck up and blow up their cloud services or have them detonate that are completely innocuous. occam's razor is not always applicable, but i'm pretty comfortable applying it here.
$FB and $AMZN are tanking in kind, bringing the NASDAQ and larger US economy down with it today.
Smells like panic selling to me. Big tech is too powerful to get split up by the government like this.
The S&P is a far better indicator for the "larger US economy" and it's barely down right now. I don't think we should be running around talking about the US economy coming down today.
We can definitely talk about a stagnating US economy in the past 6, and next 18, months.
That outage yesterday smelled a bit fishy. I wonder if someone's in panic mode, cleaning up data, and trying to implement new systems/algorithms in a hurry to get ahead of this probe.
That seems like a stretch to me. The outage was disastrous, I can't imagine a single higher up at Google (especially engineers) that would have been comfortable tanking all of their cloud customers and their own services for hours, nor can I imagine how that could possibly be related to this probe.
My only point is that hasty updates make a mistake. Most outages are not existing systems caving in - you can monitor for that and get ahead of it, so it rarely happens. What takes things down is making changes that you haven't thoroughly prepared for and tested. Even then, mistakes happen, behaviors emerge in the system that can cause it to lose coherence. The more complex, the more likely.
When I see an outage, my immediate reaction is to ask, "What'd you change?" not "What broke?"
Being related to the probe is speculative, but outages being a result of changes is the rule, not the exception. Haste makes that more likely. If Google is being probed, and is feeling like they need to make changes because of it, that could have made them hasty.
On the other hand, someone could have tripped over the wrong cable. That also happens. If youtube's systems start exhibiting new behaviors, I'll be satisfied it was a rocky update. They've been making a lot of changes in the last couple of months, many of the channels I follow are talking about this. They mention videos several years old suddenly getting demonetized, changes in what is advertised in front of their channels, new patterns in the ranking and recommendation systems, plus changes in what's acceptable in chats, all in the last two or three months.
The 'conspiracy' theory I hear is that youtube is gearing up for the election. I wouldn't go that far. Sounds more like google is taking the tech/social responsibility criticism to heart and is starting to make changes to deal with the issues. That's a good thing.
Edit: Wanted to acknowledge that I definitely mis-/over-interpretted your initial statement, so thanks again for explaining further.
Thanks for breaking down your thoughts a bit more. I think there are two reasons that I'm skeptical of even that viewpoint:
I agree that this was probably related to some production change in GCP, but GCP sees so many changes every day. The timing here might be making this seem substantially more interesting than it is.
I'll be reading their incident report with interest, that's for sure. Google doesn't typically make mistakes like this, which is why I'm curious - as you pointed out, it's an expensive mistake and one to be avoided at all costs.
So far, this seems to have been a routing issue according to the status pages and some comments on HN. That's typically the result of a segment going down, forcing traffic to go elsewhere, and resulting in a traffic jam when the routers get overloaded. It probably wasn't a physical outage (such as a drunk backhoe operator) because they got it back up rather fast - those usually take longer to resolve. I'd like to know what precipitated the network issues.
can we please as a collective not be weirdly conspiratorial like this when there's literally no basis for the conspiratorial thinking? cloud services explode like they did yesterday not-infrequently, and it probably has fuck all to do with anything being reported here.
It's hardly a conspiracy. Outages like this correlate to many of youtube's backend algorithm updates and have for years - not all of them, of course. It's just that rejiggering the backend does lead to disruptions in any complex system, that's how it goes in tech. It's hardly unique to google, either.
i would say it is fairly conspiratorial and that there's really no business even going down that road right now, considering that we're not even certain on whether or not a probe is actually happening, to say nothing of the total lack of evidence pointing anywhere in the general direction you're pointing here. google has a billion moving parts and a billion services, and there are a lot of reasons they could run into anti-trust issues that have literally nothing to do with things they'd need to change on their backend, and equally as many reasons that they could randomly fuck up and blow up their cloud services or have them detonate that are completely innocuous. occam's razor is not always applicable, but i'm pretty comfortable applying it here.
I'd agree with that. All I said was, "I wonder" - I wasn't expecting the inquisition. I think it's a perfectly valid question.