How do you manage data backups?
Hi Tildes. Hopefully this thread will be both a good discussion and helpful to some of you, and hopefully me.
As I'm guessing most of you know, data backups are quite important and it is best to have at least one copy locally and another copy somewhere else. At the moment, I store photos on an external hard drive and Google Drive, photos from my phone on Google Photos with copies of important original quality files saved locally, and everything else on drives in my PC and a network drive on my Raspberry Pi. It's far from ideal, I've only got one copy of some files and three or four of some others so I've been looking for something better to keep everything organised, safe and in one place.
I've tried the free trial of Backblaze, which seemed the obvious choice, but it had a few problems. I couldn't backup my Pi's network share, and in general it's a bit clunky and difficult to use. It is marketed as an easy solution to backing up data, but in doing this it just makes everything more difficult, at least for me - I know what I want backed up, and I would prefer to select it manually, but by opting in everything for backup by default you have to spend ages excluding the folders you don't want saved, one-by-one, in a UI that is difficult to use and often unclear. Sometimes the exclusions list just doesn't work - the Program Files folders are meant to be excluded by default and they were listed under exclusions but were backing up anyway. For me it found over 200,000 files, and because they were all so small it barely managed to backup 100MB in three hours. (Not that I know where the files come from because they aren't listed in the Windows app in any vaguely comprehensible way.)
So I need to find something else, and I was hoping someone here would have some recommendations. Personally I need it to:
- Be affordable and easy to setup and use
- Backup external and network drives to the cloud (physically keeping another drive somewhere else isn't an option for me)
- Be trustworthy and have strong commitments to security and privacy
- Work well for my use case: preferably automatic from Windows
Looking forward to any comments or recommendations. Thanks!
I'm sure other folks will have some really interesting set ups, but I can attest to pCloud's online cloud service. If you need less than 2 TB, this may be a good option.
Pros:
In this way, you can select arbitrary folders to sync to pCloud that may contain actual backup images, as well as regular files and folders, say from drive A, B, and C. You could then add one primary sync for an aggregated pCloud folder to another hard drive, say drive D, and basically funnel your data up to pCloud, then back down to a drive, creating parity between a physical and cloud copy.
The con of this is that these are syncs, so messing with so-called drive D will impact data in your cloud.
Wow, lifetime subscriptions. Honestly, that makes it way more desirable to me than any other cloud storage provider. But it's a bit sad that they only offer a maximum of 2TB.
pCloud can get a tiny bit pushy with upselling, so if you’re on the 500 they’ll nudge you upgrade to the 2TB, and if you’re on the 2TB they’ll nudge you to pay for the encryption add-on. Given that they’ve been maxed at 2TB for a while now, I fully expect they’re going to raise the cap and try to upsell that to their existing lifetime customers.
This is pure speculation on my part, so don’t count on it or anything, but I’m expecting to see that come down the pipeline at some point.
I like the look of that. I did think it was a little expensive at first but Google One is only £1 a month less so it's similar. I'll take a look at the free option, thanks for the suggestion.
I have two offsite encrypted backups. One at my work, another at a safety deposit box at the bank. I tried storing one backup at my sisters, but my bro-in-law threw it out.
I use BorgBackup [1] with rsync.net [2], which I use for daily/weekly/monthly snapshots. As a plus, the snapshots are also encrypted.
I think I'm paying $18/yr for 100 GB of storage.
[1] https://github.com/borgbackup/borg
[2] https://www.rsync.net/products/borg.html (Note that there's special pricing available for borg/technical users.)
all laptops & desktops (running either Mac OS or Linux) sync their home directories to a home server (running Ubuntu) via Syncthing. This is sync, not backup, but it supports versioning, which I employ.
The server, along with all user home directories, is backed up to Backblaze B2 using Duplicati (front-end for duplicity). Duplicati is great -- it's a block-based de-duplicating backup program, and I highly recommend it. Supports many different cloud storage targets, and pre-egress AES-256 encryption is supported out of the box.
I've also been very happy with Backblaze B2 -- it's crazy cheap. $0.005 / GB / month for storage (half a penny per Gigabyte per month).
I should add, all of the above is open source, and works on Windows.
B2 + Duplicati is looking like the best option so far, for me it would work out to less than £2 a month which is great!
I've been using Code42/Crashplan for quite a few years now - never had any issues. They are a bit pricier I think, but I also back up work/development stuff, so I'm ok with it. I use their cloud service, but I also have the software backup to a local hard drive as well. You can set it up to retain certain files for longer, and also specify how often you want certain files/folders to be backed up, along with how many different versions of those files you want to have.
And like you, my photos are also stored on Google photos.
It is a little more money, and from what I can see it is more business oriented. I think I remember seeing they used to offer plans for individuals but stopped that a little while ago, so are you still on one of those plans somehow or is their "for small businesses" slogan just marketing and it's actually suitable for people like me too?
They did used to have personal plans, but they transitioned to business oriented. My personal plan got rolled over, and I think I ended up with a bit of a discount. It probably depends on how much customization you need as to whether they're the right option for you. If you just need basic backup, there's certainly cheaper options.
Isn’t Crashplan only available for business now?
I don't really heavily concern myself with backups, because there's not a lot of data I care about saving, and have been successful in the past recovering data from dying machines/drives. I use ZFS almost everywhere, and have two drives on my main machine - one is a 1TB SSD and the other is a 10TB HDD. I keep a copy of important data on both, and very important data on drives on other machines. On my chromebook that has been converted to linux I keep two btrfs partitions on the same drive in a RAID1 configuration, so if it starts failing I'll be alerted early enough to pull data off it (this has happened before).
whats the reason you use ZFS/btrfs? or what can it do that ext4 can't? i tried both some time ago and did not really feel or see any difference. and if they are better, why do most distributions still use ext4?
Honestly it's mostly for the checkpointing functionality. On top of that, ZFS/btrfs are more well suited to flash storage than ext4 (to my knowledge) - but there's also F2FS if you want a filesystem similar to ext4 for flash. One last major benefit is that you don't need to know the size of different partitions in advance if you want to do things like only run checkpointing on your home directory.
Edit: Oops, forgot to answer the why don't distos use ZFS/btrfs: ZFS is licensed in a way that makes it so it cannot be distributed with the Linux kernel, and btrfs has a history of lacking stability. I actually have experienced btrfs instability myself - I much prefer ZFS.
thank you for your answer! I dove a bit into filesystems and stuff and zfs looks very very interesting, but it seems to be a bit of problem when using a linux distro, especially when its a rolling one. what do you run it on?
I run it on the NixOS unstable branch (which is rolling) which packages it in a way that (I think) is complaint with the license.
Edit: If you start using ZFS make sure you use something like fio to test your sector size and set it correctly on the ZFS pool.
I removed the hardware write switch and flashed a new BIOS from here. After that, I installed Arch Linux. A bit more information in this topic.
I use http://duplicity.nongnu.org/ .
Can do encrypted back up to any local filesystem location, or remote via ssh/scp, or any of the popular cloud storage services (S3, Azure, etc.). Free/libre open source.
Duplicity has some inclusion and exclusion filtering features, but what I do is put everything that I want to be excluded from backup into one directory. That way, I only have to pass one thing for exclusion to duplicity, and not think about picking lots of things to include or exclude.
That was my problem with Backblaze, there isn't an option to pick things to include, only pick to exclude, and I definitely can't move all my windows system folders into one! Although my list of things I want to include is quite small, everything on a couple of my drives and only a few folders on the other two. Someone else said Duplicati is a GUI for Duplicity, that's looking like the best option so far. Thanks
I back up my desktop computer using Time Machine locally and Backblaze for offsite backup. Backblaze is not the perfect solution but better than all others I've tried and easy to set up and reliable, and that's what I want from a backup solution.
For my Server and/or Raspberry PI I'm using restic and back up to Backblaze B2.
Over the years I tried all sorts of tools and services like Google Drive, unlimited OneDrive and so on, but I think one's better off just having a solid and reliable setup. When you read up on r/DataHoarder and find out that some "unlimited" or "lifetime" service goes out of business or wants to charge more money it's a huge effort to move things around.