Shopping around for a new-and-improved backup solution

I'll preface this by saying I may have done something wrong, but I originally used Borg to backup my NAS to an external drive. NAS mostly holds backups of my BR disks and backup images from the family PCs. It was fine until the repo approached the size of the disk. Pruning wasn't enough to trim the repo size down, so I was stuck in a loop where the backups could no longer complete, and I could not trim the repo size down further. These attempts at pruning would sometimes take multiple days. There were a couple of other "gotchas" that affected this workflow that I can't fully recall at the moment, but I ultimately had to nuke the backup after a few years and start over. I went with Backrest which is a webui for Restic. I have had no similar problems. The repo trims itself as expected and seems to work quickly. I tested file recovery and I was able to retrieve the files without drama.

Keep in mind I am using this to backup a NAS - my root file structures are not changing often, if ever. New directories will occasionally be added to the root, but I am not going to the root level and moving things around.

I am using restic through backrest which just makes it slightly more convenient with a visual overview. It all goes to a hetzner storage box.

Or can I navigate into the repo and just manually grab files?

I can choose what files/folders specifically to restore/download as well from each snapshot.

As far as the parent directory changing goes. I guess that might be an issue, but I don't do that very often. Certainly not for directories that contain large files. Movies and series are sitting on my NAS and aren't backed up, development stuff sits in git repos and doesn't need to be included. So my restic backups are fairly consistent in size.

I used rclone for a while to back up to cloud providers, but I am currently between solutions. Rclone might not tick all your boxes but I was pretty happy with the experience. The only reason I'm not using it now is because I was relying on unlimited cloud storage and I haven't yet decided on a solution that doesn't.

I've been using Plakar for a while and am pretty happy with it. I have backups run at different frequencies based on the needs of each set of data, and then back up the most frequent snapshots to a cloud S3 provider on a less frequent basis.

It took me a while to figure out how to set it up the way I wanted, but it's been chugging along in the background now with no issues.

I've been on the free version of Veeam for over a decade, and it has been fantastic the few times I've needed it, both for individual file restoration, and volume level restoration. My only complaint is that I have to use pre/post backup scripts to back up remote machines over sshfs, as the native options are a local filesystem, samba share, or a VB&R server (not free).

Hi @Eric_the_Cerise
So, first of all, for a number of years, I used to just manually copy/paste my home directory over to a samba destination drive/share on my local LAN...and then also copy that to one of those external USB drives....Slow as hell, and cumbersome, but if all things failed except for the USB drive, then it would be easy - though slow - to recuperate things. Its a pretty inefficient method, but pretty good...well, it depends how much data, speed of the USB ports, the LAN/network, etc. But, it was solid, did not require special apps/clients, and as noted quite direct to restore things.

Then, I started using rsync (and very, very similar parameters to the command you referenced above, though i exclude other stiff not videos)...After an initial copy/paste of a directory, then I just keep using rsync at a regular interval, and its worked GREAT! The delta approach that rsync uses (only sending changed stuff) is so powerful! Yes, other apps, clients also use delta transfer, but nothing so non-specialized as rsync. Are there better backup systems? I'm sure there could be. But, you should consider, what is it that you are trying to improve, what are you hoping to achieve by reviewing other approaches, apps, clients, etc.? Its possible what you seek can be better implemented by something other than rsync...but when i reviewed my needs, rsync met them way better than anything else that i have tried...Everyone's mileage will vary. :-)

I am using restic but I evaluated borg too. I can't remember now why I went with restic, it may have been that restic is written in go and borg is python. Python does not feel right for something like backups. Restic is a bit fussy to set up though, I have a little script that mounts the archive and that makes it not too bad.

Have you looked at rsnapshot? It is written in Perl so perhaps from that point of view that's worse than Python, but it just works. For one of my backups I use rsnapshot to take a daily, weekly and monthly snapshot. It uses filesystem hard links so if you have a lot of immutable files the snapshots don't take up extra disk space. And because it is just using the filesystem it means all your files are just there to access, no software needed for recovery.

One to possibly add to your list is duplicati. Cant speak to all of the pros and cons but I was using it for a long time for off-site backup to a small low power NAS that only supported FTP. The software also supports a whole bunch of other data transfer methods but fully runs as a client only application. Deduplication is managed at a block level so it should manage your folder change example gracefully. It will see the move as a new large file but will reuse the old blocks from the previous location meaning it will not create new ones.

I setup the remote NAS to connect to my network as a VPN client and act as a ftp server. Weekly I would pause all of my docker applications and copy app data to a staging folder. Duplicati would then process the app data as well as normal file data for changes and upload. It's not super fast but neither is the remote NAS.

I was happy with how simple it was to configure and run it in a docker container. If you already have a docker setup I would recommend pulling down an image and poking with it as it was fairly simple to get going quickly.

I would love to add to this discussion but I just have calendar reminders to copy my "server" folder over to my "backups" HDD manually every week and upload a copy of everything in "server" except for music and movies to my cloud account. So I will read through the replies and try to further my education.

duplicity has been largely set and forget for me. I have it writing to an external HD which I have in mind to disconnect and take in the event of an emergency. Automatic full vs. partial backups, and rotation/drop of old backups. I'm sure it could be set up to be part of a remote backup setup if one wanted.

The fact that merely renaming a parent directory (or grandparent, or great-grandparent, etc) causes restic to treat every unchanged byte below that as brand new ... that's a recipe for giant, bloated repos, and it's unacceptable to me

I believe that Restic stores all the files in a content-addressed manner where the contents will be deduplicated even in this case. When it encounters file paths it doesn't recognize or files with an updated last-modified timestamp, it has to re-read them but it will end up recognizing that their contents already exist in the backup if they're unchanged.

I wouldn't worry too much about the possible redundancy of your backup's encryption when using a tool like Restic. If your backups are on a filesystem that's already encrypted and you're worried about the risk of losing the Restic key, you could always save that key in plaintext right next to the backup. And if you do ever move/copy your backups off that encrypted filesystem onto an unencrypted system or a cloud host, it will be convenient to be able to move/copy the backup files as-is instead of needing to layer in another program to manage encryption.