When it went down I imagine a lot of sysadmins showed up in the parking lot. I've been in that crowd. You get the text messages from your automated services telling you something is down so you...
When it went down I imagine a lot of sysadmins showed up in the parking lot. I've been in that crowd. You get the text messages from your automated services telling you something is down so you pop in at the data center promptly to fix your machines. It never occurs to you that everyone is having that problem due to a major service outage like this fire (or in my case, the main fiber line being cut) and before you know it there are 30 sysadmins talking shop in the lobby.
They'll find out who has the write-only backups and not get much sleep in the next week. This would have taken me 2-3 days to recover from, most of it waiting on new hardware. This is why business continuity insurance is a good idea.
The fun part is the bits that aren't in the traditional backups, easy to overlook - router and firewall configs, for example. There's an awful lot to miss when the entire hardware stack gets reduced to embers.
Update:
We've confirmed a total loss of the affected EU servers during the OVH data centre fire. We're now exploring replacing the affected servers.
Data will be unable to be restored.
I don't play Rust, but I saw this earlier this morning.
Is the building built out of shipping containers? Is that a common practice? It is probably hard to fight fires in such setup compared to a regular datacenter building.
Is the building built out of shipping containers? Is that a common practice?
It is probably hard to fight fires in such setup compared to a regular datacenter building.
Yep. See here for more info: https://web.archive.org/web/20170609052044/https://www.ovh.co.uk/a656.SBG1_central_europe_data_centre I wouldn't necessarily call it common, but it's not a...
I've been in IT for 25 years, and the joke has always been "back it up, in case it catches fire". Well, I've seen (and done) many stupid mistakes, mishaps, disasters and unfortunate events...
I've been in IT for 25 years, and the joke has always been "back it up, in case it catches fire". Well, I've seen (and done) many stupid mistakes, mishaps, disasters and unfortunate events mitigated by back-ups (or made worse by lack thereof), but until now I had never seen a computer actually catch fire.
Now, the much touted cloud has become a proper cloud of smoke.
When it went down I imagine a lot of sysadmins showed up in the parking lot. I've been in that crowd. You get the text messages from your automated services telling you something is down so you pop in at the data center promptly to fix your machines. It never occurs to you that everyone is having that problem due to a major service outage like this fire (or in my case, the main fiber line being cut) and before you know it there are 30 sysadmins talking shop in the lobby.
They'll find out who has the write-only backups and not get much sleep in the next week. This would have taken me 2-3 days to recover from, most of it waiting on new hardware. This is why business continuity insurance is a good idea.
The fun part is the bits that aren't in the traditional backups, easy to overlook - router and firewall configs, for example. There's an awful lot to miss when the entire hardware stack gets reduced to embers.
https://twitter.com/playrust/status/1369611688539009025
I don't play Rust, but I saw this earlier this morning.
Is the building built out of shipping containers? Is that a common practice?
It is probably hard to fight fires in such setup compared to a regular datacenter building.
Yep. See here for more info:
https://web.archive.org/web/20170609052044/https://www.ovh.co.uk/a656.SBG1_central_europe_data_centre
I wouldn't necessarily call it common, but it's not a particularly new idea, nor is OVH the only one that does it. See: Modular data center, Shipping container architecture, etc.
There is an unusually relevant xkcd for this event. ;)
I've been in IT for 25 years, and the joke has always been "back it up, in case it catches fire". Well, I've seen (and done) many stupid mistakes, mishaps, disasters and unfortunate events mitigated by back-ups (or made worse by lack thereof), but until now I had never seen a computer actually catch fire.
Now, the much touted cloud has become a proper cloud of smoke.