This is specifically looking at contracts they were in the process of closing on. There may be others who choose not to renew in the next one or two cycles, though that has costs and they would...
This is specifically looking at contracts they were in the process of closing on. There may be others who choose not to renew in the next one or two cycles, though that has costs and they would need to identify a replacement.
Most companies also try and maintain an accounting of "goodwill" which is basically rolling up brand recognition, trust, and other intangible reputational assets. Those assets are surely heavily discounted due to this.
There was definitely a vibe on Reddit that is still the best company at what it does. Also changing a software vendor, especially if you have 100k+ computers has its own cost involved.
There was definitely a vibe on Reddit that is still the best company at what it does. Also changing a software vendor, especially if you have 100k+ computers has its own cost involved.
I don't know how many endpoints at my work are using Crowdstrike, but it's definitely in the 10s of thousands. The water cooler chat is that there are plans already being enacted to move away from...
I don't know how many endpoints at my work are using Crowdstrike, but it's definitely in the 10s of thousands. The water cooler chat is that there are plans already being enacted to move away from CS over the next few years.
I still can't process that not a single person tested this updated before they published it. No manual tests, not automated, nothing. I know developer are lazy, mistakes happen, I'm a dev. That's...
I still can't process that not a single person tested this updated before they published it. No manual tests, not automated, nothing. I know developer are lazy, mistakes happen, I'm a dev. That's why we have automation and processes in place to prevent this kind of thing. Especially considering how critical their products are, this sounds super amateurish.
I can't speak with 100% certainty, but I thought I had read/heard that it did pass initial QA testing but something in their release pipeline caused the channel file to be nulled out which was...
I can't speak with 100% certainty, but I thought I had read/heard that it did pass initial QA testing but something in their release pipeline caused the channel file to be nulled out which was what led to the meltdown.
Still doesn't excuse not testing the release package before pressing the "upload to all customers" button. We have sanity checks for a reason lol. Complacency is the enemy of sanity and sounds like there is some rot in the company that needs to be worked out
The biggest failure is that there's an "upload to all customers" button, at all. For the level of business they do, they ABSOLUTELY should have a slow and phased deployment plan for all changes....
The biggest failure is that there's an "upload to all customers" button, at all.
For the level of business they do, they ABSOLUTELY should have a slow and phased deployment plan for all changes. Something like 10% of customers, wait a day, 20%, wait a few days, 50%, wait, 100%.
This is basics for any large level deployment, let alone across such a multitude of configurations and languages.
Ehh, considering they are providing security software/services, there is some justification for system of immediate release to all clients should there be an severe security risk that needs to be...
Ehh, considering they are providing security software/services, there is some justification for system of immediate release to all clients should there be an severe security risk that needs to be addressed as quickly as possible.
Not really? Even in those supposed cases they are the vast minority, not the standard, and should be required to be flagged as such and go through extra review. Even still a deployment plan needs...
Not really?
Even in those supposed cases they are the vast minority, not the standard, and should be required to be flagged as such and go through extra review.
Even still a deployment plan needs to be in place that doesn't just hit all customer systems at once because the last thing you want to do is exactly what they did. Even if this was patching some 0 day, if the outcome is the same, congrats you've probably done more damage than it ever would have if you rolled out over 3 days.
According to Crowdstrike, the null file wasn't the problem directly (idk if we've been updated on where the actual source of the error was according to them). In their report after the disaster,...
I can't speak with 100% certainty, but I thought I had read/heard that it did pass initial QA testing but something in their release pipeline caused the channel file to be nulled out which was what led to the meltdown.
According to Crowdstrike, the null file wasn't the problem directly (idk if we've been updated on where the actual source of the error was according to them). In their report after the disaster, they said that the update passed a recently-implemented automated test, but a bug in the test caused it to pass despite the problem. They definitely did not have manual QA testing of this update prior to the release.
Ah ok that makes much more sense. I haven't read their RCA yet (if it's been released to the public), just the few pieces that I picked up here and there from various people more involved than I...
Ah ok that makes much more sense. I haven't read their RCA yet (if it's been released to the public), just the few pieces that I picked up here and there from various people more involved than I in the week after the meltdown.
Thank you for the correction/information, fellow spark :)
If I understood their Preliminary Post Incident Review (PIR) correctly, they said they will implement all kinds of testing, including "Local developer testing" that I assume is asking their devs...
If I understood their Preliminary Post Incident Review (PIR) correctly, they said they will implement all kinds of testing, including "Local developer testing" that I assume is asking their devs to test it. They published later the full report but I haven't read it. IIRC they only had automated testing to check the format of the file, but that failed.
Semantic nitpick, they are actually fully compliant by not doing business in the EU and actively blocking people from there. Likely privacy violating assholes for anyone outside the EU, but GDPR...
Semantic nitpick, they are actually fully compliant by not doing business in the EU and actively blocking people from there.
Likely privacy violating assholes for anyone outside the EU, but GDPR compliant assholes nonetheless.
They wouldn’t be compliant if they were to offer their site in the European Economic Area, which is what I meant. But obviously, they can stay compliant by blocking an entire region, you’re right...
They wouldn’t be compliant if they were to offer their site in the European Economic Area, which is what I meant.
But obviously, they can stay compliant by blocking an entire region, you’re right in that. (;
I'm shocked it's not higher. I can't imagine ever wanting to use a company that bungled things this badly ever again.
This is specifically looking at contracts they were in the process of closing on. There may be others who choose not to renew in the next one or two cycles, though that has costs and they would need to identify a replacement.
Most companies also try and maintain an accounting of "goodwill" which is basically rolling up brand recognition, trust, and other intangible reputational assets. Those assets are surely heavily discounted due to this.
There was definitely a vibe on Reddit that is still the best company at what it does. Also changing a software vendor, especially if you have 100k+ computers has its own cost involved.
I don't know how many endpoints at my work are using Crowdstrike, but it's definitely in the 10s of thousands. The water cooler chat is that there are plans already being enacted to move away from CS over the next few years.
I still can't process that not a single person tested this updated before they published it. No manual tests, not automated, nothing. I know developer are lazy, mistakes happen, I'm a dev. That's why we have automation and processes in place to prevent this kind of thing. Especially considering how critical their products are, this sounds super amateurish.
I can't speak with 100% certainty, but I thought I had read/heard that it did pass initial QA testing but something in their release pipeline caused the channel file to be nulled out which was what led to the meltdown.
Still doesn't excuse not testing the release package before pressing the "upload to all customers" button. We have sanity checks for a reason lol. Complacency is the enemy of sanity and sounds like there is some rot in the company that needs to be worked out
The biggest failure is that there's an "upload to all customers" button, at all.
For the level of business they do, they ABSOLUTELY should have a slow and phased deployment plan for all changes. Something like 10% of customers, wait a day, 20%, wait a few days, 50%, wait, 100%.
This is basics for any large level deployment, let alone across such a multitude of configurations and languages.
Ehh, considering they are providing security software/services, there is some justification for system of immediate release to all clients should there be an severe security risk that needs to be addressed as quickly as possible.
Not really?
Even in those supposed cases they are the vast minority, not the standard, and should be required to be flagged as such and go through extra review.
Even still a deployment plan needs to be in place that doesn't just hit all customer systems at once because the last thing you want to do is exactly what they did. Even if this was patching some 0 day, if the outcome is the same, congrats you've probably done more damage than it ever would have if you rolled out over 3 days.
According to Crowdstrike, the null file wasn't the problem directly (idk if we've been updated on where the actual source of the error was according to them). In their report after the disaster, they said that the update passed a recently-implemented automated test, but a bug in the test caused it to pass despite the problem. They definitely did not have manual QA testing of this update prior to the release.
Ah ok that makes much more sense. I haven't read their RCA yet (if it's been released to the public), just the few pieces that I picked up here and there from various people more involved than I in the week after the meltdown.
Thank you for the correction/information, fellow spark :)
If I understood their Preliminary Post Incident Review (PIR) correctly, they said they will implement all kinds of testing, including "Local developer testing" that I assume is asking their devs to test it. They published later the full report but I haven't read it. IIRC they only had automated testing to check the format of the file, but that failed.
Anyone have a mirror? Content is not available in my region apparently. :(
Changed the link to the original source, Associated Press.
Thank you!
Wow, it’s been a few years since I’ve last encountered a site with such blatant non-compliance with regard to (EU) GDPR…
Semantic nitpick, they are actually fully compliant by not doing business in the EU and actively blocking people from there.
Likely privacy violating assholes for anyone outside the EU, but GDPR compliant assholes nonetheless.
Many GDPR provisions do cover EU citizens living abroad, so geoblocking is not a way to be "fully compliant" with it. It's wishful thinking.
They wouldn’t be compliant if they were to offer their site in the European Economic Area, which is what I meant.
But obviously, they can stay compliant by blocking an entire region, you’re right in that. (;
It's been changed to a better link.