30
votes
What is the most reliable and affordable form of storage medium to use as a backup drive for your computer?
I just had my backup hard drive die and while it did last a few good years, I just want to know what everyone else is using and what gets the best bang for buck.
This depends a great deal on how much data you have to back up.
Also, obviously, your use cases will play into this; if you need your backups to be online or regularly-updated, then you're looking at a NAS or cloud storage solutions. A single external backup drive can work, if you regularly test it to make sure it's still readable.
LTO drives cost thousands of dollars, I think you would need to get to hundreds of terabytes of backups before they start to seem feasible.
True. I failed to mention that the drive is something I would throw in a go bag in case of a major disaster and not necessarily something that would be my go to for a data restore.
Personally I use an external HDD for work backups and cloud for what little personal stuff I need kept safe. But if you want maximum reliability you'll want optical media. You could get re-writeable discs, but if your needs are mostly for new files (like storing raw photos and videos) and don't need to update anything then single-use would be a perfect fit. Blu-Rays can hold as much as 100GB per disc. It will be more expensive than an HDD, though.
Interesting, yea I've often wondered what would be a bomb-proof method of storing a backup of my photo collection, and maybe bluray discs are the way to go. I have CDs from the 80s that still function just fine.
Do you think there is any disadvantage of rewritable vs non-rewritable? Are rewritable any more vulnerable?
Worth noting that those old CDs are going to be stamped, whereas anything writable at home uses dye, which changes the longevity significantly. That said, if you keep them in the fridge it’s reasonable to expect a few decades (the Library of Congress has done some somewhat dated but highly relevant research on optical media ageing).
Beyond temperature and humidity, the other key factors are storage density (Blu-ray will likely degrade to the point of error more easily as the physical features are much smaller) and precise manufacturing/chemistry (they saw some notable bad batches that almost all degraded far faster than the average).
Personally I prefer a few copies on actively spinning hard drives - NAS at home backed up to a cloud provider - but if you wanted to go optical I’d imagine at least two copies of everything on media from different manufacturers would be a sensible way to go about it.
Tape might also be an interesting one: it’s still the way big providers do large, long term cold backup, and the price per TB is hard to beat, but the setup costs are normally prohibitive for home users.
Print your photos into photo books. The smarter version of myself does this each year.
Years ago I did the math on how much I was paying to replace external storage and it’s reliable. I decided it was basically a wash to pay for online storage for anything I couldn’t replace by reloading software. There are plenty of options out there depending on your tastes. Personally I just use OneDrive with the quantity of items I have on my PC. It cost me 70 bucks a year and includes the whole Microsoft 365 suite. So that’s mostly documentation and some archive photos. I’m in the Apple ecosystem for most of my photography as well as my spouse and we used shared iCloud storage for photos and miscellaneous files on those devices. I use gaming cloud services for game saves on pc and consoles. One factor I considered highly was that anything that would take out my devices would likely destroy any onsite backups I would have. I just felt more comfortable with the trade offs of offsite/cloud storage. I have recovered my pc and migrated new devices more than once and the only thing really limiting has been the bandwidth.
Well I suppose rewritable discs could possibly be accidentally overwritten. They also might be slightly more expensive.
Does age of the HDD matter? I have a 15 yo backup HD that still runs fine (albeit slowly) and of course the size is an issue at this point, but should I retire it completely and buy a new HDD? Are there particular brands that are recommended?
Yeah, hard drives eventually just wear out, they’ve got a lot of physical moving parts that can fail over time.
They tend to follow a bathtub curve: higher failure rate when nearly new from manufacturing issues that QA didn’t catch, low chance of failure for a few years after that as the bad ones have already weeded themselves out, and then increasing chance of failure again as motors and bearings age, cumulative vibration damage takes its toll, etc.
Backblaze publishes quarterly reports on reliability at scale, but on an individual level when you’re buying one or two of the things it’s pretty much luck of the draw. The major manufacturers all make decently reliable products, some will be DOA, and all will fail eventually - often at the most inconvenient possible time - so you’ll want to plan accordingly.
Would an external HDD be a smart solution? My understanding is that an SSD will eventually break (or something) if not in use for years (and I think even they'll wear out eventually in use?) but are HDDs pretty much forever safe if stored properly?
If the drive is just standing on a shelf, would it be smart to buy a new one every five years? Or ten? Or am I totally off the mark?
My take on your question is that if you have an HDD that is just sitting on a shelf untouched, even if it's more reliable than an SSD in that situation, you have absolutely no guarantee that it's going to work when you need it to.
You need to test your backups. My approach here is that cloud providers can't compete with the cost of an 8TB SATA drive that's brought home from the office once a week and rsynced, then taken back to the office. Every week I plug that thing in and I know that it works.
It’s all a question of risks and probabilities, really. Keep a spinning disk powered on, eventually it wears out. Keep it off, eventually the bearings seize. Write too many times to an SSD without realising, the NAND degrades. Unplug it for cold storage, eventually the charge in the cells dissipates. You’re generally better off planning for hardware failure, rather than trying to avoid it and ending up screwed if you get unlucky.
I’ve seen conflicting things on whether keeping drives spinning may actually be safer than storing them powered off (steady state causing consistent wear vs less overall wear but happening in an inconsistent way), but the way I’d look at it is that even if keeping them active marginally increases the odds of failure, it’s worth it to get an immediate notification when that happens rather than only finding out about it after it’s too late.
There are tens of similar trade offs for pretty much any option, and any single option will fail eventually - so it’ll always come down to your personal balance of complexity, risk tolerance, and cost. The more you care, the more copies you want, across more locations, in more formats.
As a minimum reasonable balance for the general user, I’d suggest a USB hard drive plus a cloud backup (Backblaze is a good option, but by no means the only good option). The drive will definitely fail at some point, and the chances are you’ll also realise you misconfigured or deleted something important from the cloud at some point, or get locked out, or forget you changed your card number and have it expire, or whatever - but the chances of both happening simultaneously are low enough to be acceptable for most people.
I put a lot of value on my data, so my own setup is a local NAS with two drive redundancy, backed up to a cloud bucket with point in time snapshots managed from the local side as well as a six month versioned expiry period for anything that does get deleted on the cloud side in case a whole snapshots gets accidentally wiped. That’s expensive, complex overkill for most people, but I generate a ton of code and data that’d cause me way more hassle than it costs to keep safe.
3-2-1 Method: 3 copies, 2 copies on different physical media and 1 copy off site.
Spinning hard drives (in your format of choice) will give you best value | size, most likely. Blu-ray M-DISC is good for archival. S3 cold/archive cloud storage would probably work great for the off site and is relatively cheap.
For my photo collection which is mainly historic and static I go a bit further:
Local NAS > Backed up to an external hard drive stored in another part of the house in a somewhat water/fire proof container and in my go bag stuff
Blu-Ray M-DISCs stored offsite (have used friends, bank safety deposit boxes, etc. far enough away from home in case of a local, natural disaster)
Amazon S3 cold
Invest based on what the data is worth to you and how gutted you would be if you lost it.
ZFS is an open-enough format. It's open source, just not GPL compatible.
As far as cost effective, rsync.net is the cheapest cloud storage (I had found) by a large margin.
I just mean you don’t want to try and hunt down a specific computer/OS to read a specific format of disc. Most people would be hard-pressed to find a machine to read a ZFS drive especially in a disaster. ExFAT seems to be universal enough for a portable HDD that should be able to be read by most.
Wasabi is a bit cheaper ($7/TB/month) than rsync.net ($12/TB/month) and also has no egress charge.
I suppose it depends on your recovery scenario. If you need your children or parents or any random less-techy person to be able to easily recover, then yeah, ZFS might be a barrier. But any Linux system can get ZFS installed on it pretty trivially, so I'm not sure it's a significant concern if you expect to be the one doing to recovery.
If we are talking disaster scenarios - your village has just been destroyed by several missiles, your town is under water/on fire, (insert catastrophe) and all of your data that you have access to is sitting on a ZFS formatted drive in your go bag and you don’t have a computer? And people at the shelters/camps don’t have a Linux machine?
If it’s just grannies pictures from a few decades ago and they are sitting in the cloud somewhere, probably not a major concern at the time. If you are storing documents, really important stuff that you need access to across a range of platforms without needing security credentials I would not want to be trying to find someone that can read a ZFS drive.
Absolutely. In a local or regional disaster scenario, especially where access to the data is needed during the disaster, ZFS is probably not the best choice. That's more or less what I was driving at - understand your recovery scenarios and what those requirements are.
In a situation where internet is down, electricity is spotty and artillery shells are falling a block away from your location, will you be That Guy who starts installing Arch on a random laptop? =)
Personally I can't imagine what data I would have that would be useful to actively recover before my personal safety is sorted.
I have pictures of passports and other similar documents stored, they might be useful if the physical copy is lost.
That's a great point. I might consider doing a tiered solution - stuff that would be useful during a disaster scenario on something like a couple of flash drives, but primary backups on ZFS drives.
I've never used any of these cloud storage services, but isn't Backblaze even cheaper? Their site says Backblaze B2 is $6/TB/month (https://www.backblaze.com/cloud-storage)
hetzner is still a lot cheaper.
starting at 3.50 Euro a TB
https://www.hetzner.com/storage/storage-box-mobile/
Probably also worth pointing out Hetzner minimum is 1TB, others like blackblaze you pay for what you're using so if it's only say 250GB it'll be a lot cheaper at that point in time.
Filen.io has a nice price and a free plan to try.
I guess if you're not running Linux full time it's a bigger concern. For me everything of value flows through my server for backup, so the second-tier stuff working with server matters more.
exFAT is not an intelligent filesystem though, so checksumming yourself matters more.
I like how rsync (which has borg support tier for roughly $8/TB) just gives me a mountable disk. Makes it trivial to add to any of my computers, no specialty software needed.
Personally, I didn't know what this meant (which is embarrassing, because I consider myself relatively techie for a non-programmer). but the New York Times Wirecutter has a buying guide with a brief explanation that I found helpful.
Personally, FWIW, I just have a combination of a local backup (an 8TB external HDD on my desk backed up using Macrium Reflect) and a cloud backup (a regular Backblaze account, which is somewhere in the realm of $5 a month). It's not perfect, but it's affordable and at least has an off-site component.
Cloud storage for small amounts of data you want to survive a house fire, and spinning rust for large amount of data you don't.
"most reliable" would be some kind of "enterprise" hard drive. however, those will be very expensive.
"best bang for the buck" instead will be two less expensive consumer-grade drives that you treat as identical copies of each other.
importantly, you want the two drives to be in different failure domains.
for example, what you don't want to do is decide you want an Acme 1TB SSD, then go to that product page on Amazon, change the quantity drop-down to 2, then click Buy Now.
most likely, someone at an Amazon warehouse would grab two drives from the same bin, and ship them to you in the same box.
and before they arrived at the Amazon warehouse, there's a decent chance those two drives rolled off the assembly line at the Acme factory on the same day, then got loaded into the same truck and cargo ship on their way to the Amazon warehouse.
any subtle environmental factors that can lead to a drive failing early - high heat or humidity at the factory that day, an especially bumpy road that the truck drove on, etc - are shared by the two drives. ditto any flaw in Acme's drive design that leads to a high failure rate. this means that their failure probabilities are linked together, when what you want is independence in the statistical sense.
many drives may actually come from the same factory, even if they're marketed as different brands. an easy option to avoid this is to get one SSD and one spinning-rust drive. that also isolates you from some other common failure modes (an earthquake, for example, might vibrate the hard drive enough to cause it to have read errors, but the SSD would be unaffected)
it's possible to run the two drives in a software RAID configuration (ZFS, BTRFS, mdadm, etc) as others have mentioned, but this is probably overkill. a simpler option is to treat them as two completely independent drives, and run all of your backups twice.
make sure to set up SMART health monitoring of the drives - modern drives can detect that they're starting to fail and warn you early.
if you're using backup software of some kind, it likely includes checksums of the data in order to verify integrity. if you're just copying files directly to the backup medium with
rsyncor whatever, you'll want to add checksums yourself by runningb2sumor something similar, and storing the computed checksums on the drive for later verification. make sure to actually verify the checksums regularly (at least monthly, I'd say) in order to spot any silent data corruption.It depends on what you consider reliable. Does it have to be off-site? Would you consider a cloud backup service or do you prefer external media in your home?
I personally use Backblaze which has worked flawlessly. My only gripe is that their UI when restoring is pretty bad, but seeing as I rarely use it and it still works, it’s not a major problem. For large amounts of data to restore they can even send you an HDD with the data, which is free as long as you send the drive back when you’re done with it.
I'd prefer some form of physical media kept at home. Maybe another backup at a friends house to protect against fire.
I totally get that. A cloud service is just a relatively cheap and simple way to do off-site backups, but backups that you have control over are generally the preferred option in my opinion.
I'm using RAID5 at home (NAS, home directory there, OS is on SSD in every desktop). Because I know RAID is for redundancy and not backup, I also have RAID1 in different location connected via my own VPN to which I do manual backups when I feel like it's needed. I also have one additional server with just SSD for OS and one big HDD for media (Jellyfin on VPN to stream my media anywhere) which could also be used for backup but doesn't have redundancy.
At the very least I would advice everyone to have their critical data (family photos and videos, own projects, ...) backed on one external HDD. Better yet on two such HDDs. I would use standard mechanical drive, not SSD. If possible, have one of thoe HDDs in different location (parent's house for example).
The most reliable digital storage for long term would likely be tapes stored in cool dry conditions, and if dealing with huge amounts of data tape backups become comparatively cheap. The cost of tape drives though ruins the affordability side of things for home use (and the data amounts are small) so I would - and do - go for regular old spinning hard drives for backup purposes.
This video was just released by Explaining Computers: "Storage Media Life Expectancy: SSDs, HDDs & More"
I'm gonna go a different direction than most of the other guys here. The most affordable and reliable way is to let the pros handle it. Personally I'm using S3 Glacier. Cheap as long as you dont need to recover it. I encrypt and store via duplicity.
Seconding Glacier, I have a decent chunk of data there and only pay 11 cents a month for storage. My process is currently very manual though so I will look into duplicity, thanks for the suggestion.
A RAID1 of BTRFS disks. Runs monthly scrub to validate data integrity on-disk. I use BTRFS since it's in-kernel and works excellent for hacked-together collections of old hard drives. If a drive fails, a new bigger drive gets added to live system, oldest drive gets moved to the backup system.
When backing to non-BTRFS mediums, especially CD-R, I dump a shasum (
sha256sum ./folder/* > folder.shasum) next to it, or create a torrent file, which does essentially the same but slightly harder to use. Run a scheduled job to insure the folder matches the checksum, otherwise you're suffering bit rot.This will let you validate data integrity and let you know if the backup (or raw data) is still good.