I'm just going to come right out and say it: Are we in the middle of an all out cyberwar and for some reason the government doesn't want it's people to know?
I'm just going to come right out and say it: Are we in the middle of an all out cyberwar and for some reason the government doesn't want it's people to know?
<snark>Yes, but the belligerents are giant tech corporations.</snark> It's possible, but I wouldn't bet on it at this time. The systems which have come down in the past few weeks have all been, to...
Exemplary
<snark>Yes, but the belligerents are giant tech corporations.</snark>
It's possible, but I wouldn't bet on it at this time. The systems which have come down in the past few weeks have all been, to put it politely, brittle. The Danfoss freezers might as well have been designed to be hacked, and the FAA, airlines, and banks are all well known to use extremely legacy systems. A series of state-coordinated attacks is possible, of course, but hardly the only explanation, and I think Occam suggests that this is just the "find out" phase of decades of disinvestment in security and maintenance.
(Also, given how Mullin and Duffy sniped at each other over the ATC outage earlier in the week, do you think there's any chance it wouldn't have come out if there were evidence of foreign interference?)
Something to consider: the bar for system security hovers right around the point where your system is exploited immediately when you connect it to the Internet. (Protip: don't connect your shit to the Internet. Seriously. It is basically always a bad idea.) If you make script kiddies 10% more effective—something which is well within the capabilities of LLMs—an enormous universe of almost-cracked endpoints immediately becomes vulnerable, with absolutely no changes to security posture or international relations required.
(Okay, I guess I wasn't snarking about giant tech corporations after all.)
My own armchair gut feeling theory is that we are mostly just feeling the effect of an all around acceleration in the tech industry. Everyone is just moving faster, of course mostly fueled by AI...
My own armchair gut feeling theory is that we are mostly just feeling the effect of an all around acceleration in the tech industry. Everyone is just moving faster, of course mostly fueled by AI and regardless of the quality of AI output in software, shipping more stuff faster just leads to more things going wrong on average. And of course most mistakes happen when you move too fast. There does also seems to be shift in AI tools capabilities when it comes to hacking, which has been accelerating the number of CVEs and made attacks easier to execute. I don't have any concrete numbers, but in my own work it definitely simply feels like both ourselves, our providers and our customers are having an increase in general instability and weird shit happening. More and more things are just moving faster, and we puny humans are struggling to keep up with everything.
I’m trying to keep Occam’s razor in mind for these recent events, but it’s hard to identify what the simplest explanation is for these events occurring in quick succession. A cyber attack does...
I’m trying to keep Occam’s razor in mind for these recent events, but it’s hard to identify what the simplest explanation is for these events occurring in quick succession.
A cyber attack does provide a single point to explain all these infrastructure failures and can explain why they all happened in a span of a single week.
Unrelated infrastructure failures is also plausible because a lot of our tech infrastructure is interconnected, so single failures can have a cascading effect across other services. Outages like these, except for the FAA ground stop, are not unheard of, so any single outage wouldn’t necessarily be inconceivable.
The part that is hard to explain is why this many unrelated failures would all happen in such a short span of time. Does the short time frame for these events make a cyber attack more plausible than standard infrastructure outages?
I think the most likely explanation is just that random events tend to cluster. It does feel like the failure rate has increased over the last year or so. I don't have evidence to back that up,...
The part that is hard to explain is why this many unrelated failures would all happen in such a short span of time.
It does feel like the failure rate has increased over the last year or so. I don't have evidence to back that up, but @whbboyd's and @winther's comments resonate with me. If we are moving faster and breaking more things, just in general, we should expect to see more clusters of unrelated events.
(We should also expect voids between the clusters. But "election results not widely contested" or "government not shut down like last year" don't make good headlines because news stories are about events, not their absence.)
Not just that, but when we see things, we're primed to look for and point out unrelated similar events. It's why for a while everyone was wondering why there were so many plane crashes, despite...
Not just that, but when we see things, we're primed to look for and point out unrelated similar events. It's why for a while everyone was wondering why there were so many plane crashes, despite the statistics showing that we were at a below average rate.
Yes, this is what I was going to chime in to say. Normally, no one outside those fields gives a shit about IT system reliability or cybersecurity. However, as soon as there's a high profile event,...
Yes, this is what I was going to chime in to say. Normally, no one outside those fields gives a shit about IT system reliability or cybersecurity. However, as soon as there's a high profile event, everyone notices things constantly and asks "is this part of a coordinated attack? Are these things related?"
I notice this at my job as a microcosm. We've had large scale system failures or cyber attacks in the past, and for weeks afterwards, executives from random departments who normally don't give a shit about technology and can barely manage to turn their computers on are telling me that the server failure we have today are almost DEFINITELY related to the email availability issue we had two weeks ago, because "I don't believe in coincidences".
In reality, systems are failing all the time, every day, everywhere. Most of the time you don't hear about them because they don't affect you. Once everyone is on the hunt for those stories though, they start popping up everywhere.
True to an extent. There were multiple high profile failures in a short extent though: Microsoft Outlook took down businesses across the world, then the next day the FAA was grounded, then JP...
True to an extent. There were multiple high profile failures in a short extent though: Microsoft Outlook took down businesses across the world, then the next day the FAA was grounded, then JP Morgan Chase banks had some software bug yesterday...
Yeah, it's pretty wild to have three major incidents in less than a week. Or, if we go by business days and ignore the weekend, then we've had some major system failure happen 3 business days in a row. The NSA one probably got so much coverage due to people looking for trends due to those other outages, but... Those other outages were big.
I do think the timing is a coincidence rather than cyber attacks, but it's definitely a wild one.
I was going to @ you to get your thoughts on this but I'm glad you chimed in without me performing the ritual. I really appreciate your insights into cyber security, I have learned a lot from your...
I was going to @ you to get your thoughts on this but I'm glad you chimed in without me performing the ritual.
I really appreciate your insights into cyber security, I have learned a lot from your posts on Tildes
Based on my experience over X years in the software/ systems space I would also point to the Calendar as reason we're seeing clustered failures: The underlying force is kind of ironic, but it goes...
Based on my experience over X years in the software/ systems space I would also point to the Calendar as reason we're seeing clustered failures: The underlying force is kind of ironic, but it goes like this - 1) Things are more likely to break when you change them (even if the changes are intended to make the system overall more stable) 2) Because of 1, most experienced teams avoid changing things on or about Thanksgiving/ Black Friday.... They try to rush whatever changes they want a couple weeks before (also so they can enjoy the Holiday more themselves). I don't know that this totally puts any other reason (cyber attack etc.) to sleep but it's one plausible answer to the question "why now?"
It could be as simple as one high-profile tech outage causing executives at other companies to panic a bit and mandate that their IT departments audit, update, and reinforce their own...
It could be as simple as one high-profile tech outage causing executives at other companies to panic a bit and mandate that their IT departments audit, update, and reinforce their own infrastructure to avoid similar failures.
That suddenly creates a situation where IT teams across multiple organizations are making changes to critical systems around the same time, potentially under pressure to do so quickly. And the more infrastructure you're touching, the greater the likelihood of introducing configuration errors, breaking dependencies, or triggering unexpected failures.
It could essentially create a domino effect where the initial outage indirectly contributes to subsequent, otherwise unrelated outages, simply because it prompted a wave of reactive maintenance across the industry.
And then you mix in layoffs in tech and the loss of institutional knowledge, and that increases the chance of problems even more.
Obviously it's a loose theory but 🤷. In my experience, it's sound.
There's also the possibility of heightened scrutiny in the media making subsequent outages seem more significant or connected than they otherwise would.
My bank's been down a few times in the past with absolutely no fanfare except for a temporary spike on Down Detector and never made the news.
Yet if my bank goes down in the same way tomorrow it'll be all over the news. So how many of these recent outages could just be normal?
I'm not suggesting that's necessarily what happened, but it's a plausible explanation for why several unrelated failures might cluster together without requiring a coordinated cyberattack.
I'm also pretty skeptical of how the media covers these things. Technical infrastructure failures are complicated, and I don't particularly trust general news outlets to accurately distinguish between a cyberattack, a configuration error, or an ordinary service disruption.
The media will always go with the headline that'll get the most clicks, which is cyberattack.
And frankly, if there were credible indications of coordinated cyberattacks affecting multiple major infrastructure providers, I'd expect the IT and cybersecurity communities I follow to be absolutely on fire with technical discussions, incident analyses, and people comparing evidence. I'm not seeing that level of discussion, which makes me hesitant to give the cyberattack theory much weight.
Obviously, that doesn't rule anything out, but I think there's a tendency to see several unusual events happening close together and assume they must be connected. Sometimes they are, but temporal proximity alone isn't particularly compelling evidence of a common cause no matter how many clicks corporate media wants to get. 😂
Decades of infrastructural neglect. Then large abrupt cuts to government departments that support public infrastructure or oversee private infrastructure. Then various materials become harder to...
Decades of infrastructural neglect. Then large abrupt cuts to government departments that support public infrastructure or oversee private infrastructure. Then various materials become harder to source because of world conflict. Et cetera.
These overlapping stressors build until one thing breaks. Then the complex interconnected system of civilization redistributes the load and resources in ways that are too complicated to predict, and breaks something else, and so on.
Who knows what security contractor didn't get paid to show up because the banks were down, or which bank sysadmin had their flight ground at the same time, or which leader was too distracted by one crisis to prevent another.
I think the answer is just AI. Not malicious AI, but this is what happens when really capable coding agents are set on the task of finding and patching bugs. Across the industry companies are...
I think the answer is just AI. Not malicious AI, but this is what happens when really capable coding agents are set on the task of finding and patching bugs. Across the industry companies are tightening the screws when it comes to cybersecurity. The number of new CVEs being discovered right now is sky-high, and so is the number of critical patches being rolled out on the regular now. And there has to be some urgency to it, because attackers have access to the same tools as the companies, so there’s something of a race to find the problems before the bad guys do.
In this environment it’s not surprising at all that the haste is resulting in unintended consequences. I think, or hope, that eventually things will quiet down when there are no hatches left to batten down. But I’m bracing for a lot more chaos in the coming months, before that happens.
Looking back a few weeks you'll recall that there was a big earthquake in Venezuela. I don't know anybody who could have told me how long it had been since the last tremor there, but it seemed...
Looking back a few weeks you'll recall that there was a big earthquake in Venezuela. I don't know anybody who could have told me how long it had been since the last tremor there, but it seemed like a remarkable event since there was so much footage, not only of the resulting destruction but of the destruction actually occurring. The sights were terrifying.
Within hours or days of that, footage not only of aftershocks, but of whole ass earthquakes oceans away started to appear online. I'm a huge fan of doomsday fiction and thus have to confess it was sort of scintillating to imagine that this was the big one, some kind of global tectonic collapse was about to unfold, and I'd be able to use up some of these emergency supplies.
Now its November and we've moved on to internet outages.
The people and systems who make their money by human attention have figured a lot of these quirks that we have out. They know if I clicked on 5 earthquake videos yesterday that I'm going to be hungry for new ones today.
The takeaway from this is that software development and security is a primitive science, shit is broken constantly, we're kind of where the medical science community was before penicillin or anesthesia was figured out, there is no bogeyman, and we have got a lot of work to do as an industry.
I'm just going to come right out and say it: Are we in the middle of an all out cyberwar and for some reason the government doesn't want it's people to know?
<snark>Yes, but the belligerents are giant tech corporations.</snark>
It's possible, but I wouldn't bet on it at this time. The systems which have come down in the past few weeks have all been, to put it politely, brittle. The Danfoss freezers might as well have been designed to be hacked, and the FAA, airlines, and banks are all well known to use extremely legacy systems. A series of state-coordinated attacks is possible, of course, but hardly the only explanation, and I think Occam suggests that this is just the "find out" phase of decades of disinvestment in security and maintenance.
(Also, given how Mullin and Duffy sniped at each other over the ATC outage earlier in the week, do you think there's any chance it wouldn't have come out if there were evidence of foreign interference?)
Something to consider: the bar for system security hovers right around the point where your system is exploited immediately when you connect it to the Internet. (Protip: don't connect your shit to the Internet. Seriously. It is basically always a bad idea.) If you make script kiddies 10% more effective—something which is well within the capabilities of LLMs—an enormous universe of almost-cracked endpoints immediately becomes vulnerable, with absolutely no changes to security posture or international relations required.
(Okay, I guess I wasn't snarking about giant tech corporations after all.)
My own armchair gut feeling theory is that we are mostly just feeling the effect of an all around acceleration in the tech industry. Everyone is just moving faster, of course mostly fueled by AI and regardless of the quality of AI output in software, shipping more stuff faster just leads to more things going wrong on average. And of course most mistakes happen when you move too fast. There does also seems to be shift in AI tools capabilities when it comes to hacking, which has been accelerating the number of CVEs and made attacks easier to execute. I don't have any concrete numbers, but in my own work it definitely simply feels like both ourselves, our providers and our customers are having an increase in general instability and weird shit happening. More and more things are just moving faster, and we puny humans are struggling to keep up with everything.
I’m trying to keep Occam’s razor in mind for these recent events, but it’s hard to identify what the simplest explanation is for these events occurring in quick succession.
A cyber attack does provide a single point to explain all these infrastructure failures and can explain why they all happened in a span of a single week.
Unrelated infrastructure failures is also plausible because a lot of our tech infrastructure is interconnected, so single failures can have a cascading effect across other services. Outages like these, except for the FAA ground stop, are not unheard of, so any single outage wouldn’t necessarily be inconceivable.
The part that is hard to explain is why this many unrelated failures would all happen in such a short span of time. Does the short time frame for these events make a cyber attack more plausible than standard infrastructure outages?
I think the most likely explanation is just that random events tend to cluster.
It does feel like the failure rate has increased over the last year or so. I don't have evidence to back that up, but @whbboyd's and @winther's comments resonate with me. If we are moving faster and breaking more things, just in general, we should expect to see more clusters of unrelated events.
(We should also expect voids between the clusters. But "election results not widely contested" or "government not shut down like last year" don't make good headlines because news stories are about events, not their absence.)
Not just that, but when we see things, we're primed to look for and point out unrelated similar events. It's why for a while everyone was wondering why there were so many plane crashes, despite the statistics showing that we were at a below average rate.
Yes, this is what I was going to chime in to say. Normally, no one outside those fields gives a shit about IT system reliability or cybersecurity. However, as soon as there's a high profile event, everyone notices things constantly and asks "is this part of a coordinated attack? Are these things related?"
I notice this at my job as a microcosm. We've had large scale system failures or cyber attacks in the past, and for weeks afterwards, executives from random departments who normally don't give a shit about technology and can barely manage to turn their computers on are telling me that the server failure we have today are almost DEFINITELY related to the email availability issue we had two weeks ago, because "I don't believe in coincidences".
In reality, systems are failing all the time, every day, everywhere. Most of the time you don't hear about them because they don't affect you. Once everyone is on the hunt for those stories though, they start popping up everywhere.
True to an extent. There were multiple high profile failures in a short extent though: Microsoft Outlook took down businesses across the world, then the next day the FAA was grounded, then JP Morgan Chase banks had some software bug yesterday...
Yeah, it's pretty wild to have three major incidents in less than a week. Or, if we go by business days and ignore the weekend, then we've had some major system failure happen 3 business days in a row. The NSA one probably got so much coverage due to people looking for trends due to those other outages, but... Those other outages were big.
I do think the timing is a coincidence rather than cyber attacks, but it's definitely a wild one.
This. Human nature / Baader-meinhof illusion.
I was going to @ you to get your thoughts on this but I'm glad you chimed in without me performing the ritual.
I really appreciate your insights into cyber security, I have learned a lot from your posts on Tildes
Aw well thanks, that means a lot. I try to chime in where I can about things I know about.
Some of these might be related. For example, United's issues might be a downstream effect of the FAA outage.
Based on my experience over X years in the software/ systems space I would also point to the Calendar as reason we're seeing clustered failures: The underlying force is kind of ironic, but it goes like this - 1) Things are more likely to break when you change them (even if the changes are intended to make the system overall more stable) 2) Because of 1, most experienced teams avoid changing things on or about Thanksgiving/ Black Friday.... They try to rush whatever changes they want a couple weeks before (also so they can enjoy the Holiday more themselves). I don't know that this totally puts any other reason (cyber attack etc.) to sleep but it's one plausible answer to the question "why now?"
It could be as simple as one high-profile tech outage causing executives at other companies to panic a bit and mandate that their IT departments audit, update, and reinforce their own infrastructure to avoid similar failures.
That suddenly creates a situation where IT teams across multiple organizations are making changes to critical systems around the same time, potentially under pressure to do so quickly. And the more infrastructure you're touching, the greater the likelihood of introducing configuration errors, breaking dependencies, or triggering unexpected failures.
It could essentially create a domino effect where the initial outage indirectly contributes to subsequent, otherwise unrelated outages, simply because it prompted a wave of reactive maintenance across the industry.
And then you mix in layoffs in tech and the loss of institutional knowledge, and that increases the chance of problems even more.
Obviously it's a loose theory but 🤷. In my experience, it's sound.
There's also the possibility of heightened scrutiny in the media making subsequent outages seem more significant or connected than they otherwise would.
My bank's been down a few times in the past with absolutely no fanfare except for a temporary spike on Down Detector and never made the news.
Yet if my bank goes down in the same way tomorrow it'll be all over the news. So how many of these recent outages could just be normal?
I'm not suggesting that's necessarily what happened, but it's a plausible explanation for why several unrelated failures might cluster together without requiring a coordinated cyberattack.
I'm also pretty skeptical of how the media covers these things. Technical infrastructure failures are complicated, and I don't particularly trust general news outlets to accurately distinguish between a cyberattack, a configuration error, or an ordinary service disruption.
The media will always go with the headline that'll get the most clicks, which is cyberattack.
And frankly, if there were credible indications of coordinated cyberattacks affecting multiple major infrastructure providers, I'd expect the IT and cybersecurity communities I follow to be absolutely on fire with technical discussions, incident analyses, and people comparing evidence. I'm not seeing that level of discussion, which makes me hesitant to give the cyberattack theory much weight.
Obviously, that doesn't rule anything out, but I think there's a tendency to see several unusual events happening close together and assume they must be connected. Sometimes they are, but temporal proximity alone isn't particularly compelling evidence of a common cause no matter how many clicks corporate media wants to get. 😂
Decades of infrastructural neglect. Then large abrupt cuts to government departments that support public infrastructure or oversee private infrastructure. Then various materials become harder to source because of world conflict. Et cetera.
These overlapping stressors build until one thing breaks. Then the complex interconnected system of civilization redistributes the load and resources in ways that are too complicated to predict, and breaks something else, and so on.
Who knows what security contractor didn't get paid to show up because the banks were down, or which bank sysadmin had their flight ground at the same time, or which leader was too distracted by one crisis to prevent another.
I think the answer is just AI. Not malicious AI, but this is what happens when really capable coding agents are set on the task of finding and patching bugs. Across the industry companies are tightening the screws when it comes to cybersecurity. The number of new CVEs being discovered right now is sky-high, and so is the number of critical patches being rolled out on the regular now. And there has to be some urgency to it, because attackers have access to the same tools as the companies, so there’s something of a race to find the problems before the bad guys do.
In this environment it’s not surprising at all that the haste is resulting in unintended consequences. I think, or hope, that eventually things will quiet down when there are no hatches left to batten down. But I’m bracing for a lot more chaos in the coming months, before that happens.
Looking back a few weeks you'll recall that there was a big earthquake in Venezuela. I don't know anybody who could have told me how long it had been since the last tremor there, but it seemed like a remarkable event since there was so much footage, not only of the resulting destruction but of the destruction actually occurring. The sights were terrifying.
Within hours or days of that, footage not only of aftershocks, but of whole ass earthquakes oceans away started to appear online. I'm a huge fan of doomsday fiction and thus have to confess it was sort of scintillating to imagine that this was the big one, some kind of global tectonic collapse was about to unfold, and I'd be able to use up some of these emergency supplies.
Now its November and we've moved on to internet outages.
The people and systems who make their money by human attention have figured a lot of these quirks that we have out. They know if I clicked on 5 earthquake videos yesterday that I'm going to be hungry for new ones today.
The takeaway from this is that software development and security is a primitive science, shit is broken constantly, we're kind of where the medical science community was before penicillin or anesthesia was figured out, there is no bogeyman, and we have got a lot of work to do as an industry.