30
votes
Research samples collected over decades at Swedish medical university Karolinska Institutet were destroyed when a freezer malfunctioned during the Christmas holidays
Link information
This data is scraped automatically and may be incorrect.
- Title
- Decades of research destroyed after freezer fails at Swedish university
- Published
- Feb 5 2024
- Word count
- 311 words
It feels like I keep hearing about such incidents. You'd think such high-end equipment would come with an alert module that contacts someone upon loss of power, temperature rise, or an exceptionally loud noise.
I was director of engineering for an R1 research University and oversaw network services. I always joked that the most critical devices on our network were freezers, because if they had a problem and couldn't phone home we could lose millions of dollars of research.
The freezers have modules that integrate with various monitoring and industrial control applications to report temperature and give alarm if outside of a given range. We usually dual fed the buildings with power from separate feeders and two disparate paths to the network core.
Some buildings also had generators that would kick in, and all of this with automatic transfer switches. So we took freezers pretty seriously. Additionally, when a power outage was detected, the researchers would receive alerts as well so they could manually check their stuff, in addition to facilities and the noc.
The most critical freezers were usually the -80 long term storage because they would usually have years worth of samples. We also had some odd facilities that had to be powered to maintain exact environmental controls for a large seed bank.
Very nice. That's the stuff. Regularly tested, surely?
I can't speak as much to the facilities side, but the network side we tested, and in theory the individual lab managers were responsible for testing the individual units. So probably hit or miss testing individual devices.
We had the usual complement of monitoring devices, but we also put raspberry pi's running containers to create synthetic tests inside the customer vlans so we could test inbound and out, and we tested path failures during the day to build confidence in the resilience.
I did work in operations for a while as well dealing with space, safety, capital design, etc. I will say the facilities folks did pretty good monitoring overall, and they worked closely with the utilities to deliver power on campus.
i assume the failure is most likely to be human related (improperly set up, problems with other infrastructure, lack of maintenance or poor quality maintenance, etc). it doesn't matter how fancy something is if they don't utilize it (surely there should be someone remotely monitoring every system like this, checking in like once a day for key info?? feels like there would have definitely been a huge change in temp even if some sort of other alert system failed)
I still lean slightly towards this never having been set up at all, but you present a very real possibility. Reminds me of this saying that if you make but don't test backups, you don't have backups.
EDIT: spoken too soon, you were right: https://tildes.net/~science/1e1q/research_samples_collected_over_decades_at_swedish_medical_university_karolinska_institutet_were#comment-bygi
You'd think you'd have at least two freezers
Swedish coverage:
Fryshaveri på KI – åratal av forskning förstörd
Freezer failure at KI – years of research destroyed
SVT Nyheter – 4th February 2024
We had a similar event during the big storms here last week at the marine station. The problem is that many of the freezers that this stuff is stored at -70 to -190 degrees, it takes a lot of power. The storms cut power to much of the region and their backup generator didn't work.
Many time these university setups are good, but they are still running on university budgets and can go down. In many cases all it takes is one failsafe to slip and the system is down. Surprisingly common.