What's your cloud/syncing setup for files, pics, mail, bookmarks, etc?
So I've spent the last few days trying to sync everything up between devices, with the following thoughts in mind:
- how fucked am I going to be if a device gets corrupted/stolen/lost?
- how can I easily access everything I need from a mobile device/device not belonging to me?
- how can I avoid using services from the big tech companies, and keep things open source, as much as possible?
I'm by no means an expert in the field, and I'm hoping in this thread to get a discussion going as to the pros and cons of using different services/setups, to get a general idea as to what others are doing to keep their daily lives simpler and more secure, and to perhaps see what are the future steps for me to take when I feel like playing around again.
Servers & Storage
I span up a 25GB VPS with Vultr for 'active use data', and also took out some 'deep storage'(?) from Wasabi for things which I need to keep, but not really access that much.
Protonmail with custom domain. Using the ProtonMail app for mobile, and Linux ProtonMail bridge with Evolution mail for desktop.
Nextcloud autoupload feature on mobile automatically uploads my pics to an 'autoupload' folder on Nextcloud server. Here, I categorise pics into folders and share what needs sharing before deleting anything I don't need and wiping the pics on my phone.
Nothing yet. Looking at getting KeyPass synced across devices.
Again, nothing yet. Had Firefox Sync running to connect Fennec and Firefox, but am looking for a more open approach which involves Nextcloud somehow, and allows me to tag and order things more effectively as opposed to dragging things around in the sidebar.
Evolution calendar on desktop, simple calendar on mobile, hooked up to Nextcloud and all synced using davx5
Programs and General Setup
Here, I'd like to somehow take an image/backup of my Ubuntu configs of importance and experiment with getting my setup and customisaitons replicated on another machine quickly and without taking up too much space in storage (i.e. don't need to bakckup all my files as they're already on cloud).
Also, I am very curious as to whether anybody is using Syncthing across their devices? And if so, how are they finding the experience?
I've looked into the problem of synchronizing files quite extensively.
If you're worried about losing a device, it is best to use full-disk encryption when possible. As for access from a device you don't own... that is a tricky problem that I have no solution for.
For the mere file-synchronization part, I want to mention three programs I've used/am using and their pros and cons as I see them though. Depending on your use case, one of them might fit the bill (at least for files; Calendar and Emails, etc. are a different matter entirely.)
Syncthing is basically Dropbox minus the cloud storage. Written in Go, it's simple to install and use and, from what I can tell, it is quite secure. The downside is the minus the cloud storage part. You have to have at least two machines running for the data transfer to work, and while you can shard your files into different folders that are synchronized independently, it can still be a pain to make sure everything is everywhere it is supposed to be. There is an Android app that will let you move photos off-device.
I'm no longer using Syncthing.
Keybase is currently making headlines with its XLM drop, but the app does have a built-in storage mechanism that gives you 250GB of free, (apparently) S3-backed, storage, end-to-end encrypted. It works quite well; however, as I understand it, the local devices don't hold onto the data (except for a cache), so everything usually goes across the internet. It's nice for backups or selectively sharing files with other people, but sync isn't near-instantaneous, so you can't simply use it as a Dropbox replacement. It also crapped out often in the beginning; this has gotten much better, but if it does fail, you need to notice and restart the app.
git-annex is... complicated. Complicated but very powerful. It combines git with a content-addressed storage system. The main selling-point of git-annex is that it supports tracking where what file is and how many copies you have. It also supports a lot of different backends: S3, Backblaze B2, KBFS, a plain SFTP server, whatever. Everything can optionally be encrypted. It allows you to utilize just about any file storage or sync service you can think of, including Sneakernet.
Checkouts can be partial, so you don't have to store everything everywhere -- it is very convenient to be able to
git annex drop $FILEto get rid of something large on my limited disk-space laptop, knowing that the file is still backed up in at least three other places. Similarly, if you're missing a file, it's a
git annex getaway; the program will fetch it from the cheapest available source.
git-annexcan import podcasts and download from YouTube and Bittorrent as well. It even has a dropbox-style synchronization function, the git-annex assistant. That never worked very well for me though.
The downside is that you're likely to get things wrong the first (or second) time you try to set everything up, especially if your data spans across different operating systems. Be sure to read the documentation thoroughly before you start. There also doesn't appear to be an Android app, so you need an actual computer to access your files. Windows works, but the NTFS filesystem isn't ideal for git-annex.
I have nearly the exact same setup as you and I'm pretty happy with it. I keep my KeePass database stored on my Nextcloud account. The final thing I have to do is finish setting up Nextcloud on my mini home server (Pine64 with open media vault).
It serves the purpose of being mostly open and spread out across devices. Accessing from other devices is not that easy unless you carry around a USB with e.g. you KeePass DB to access other things.
On my recently retired android phone I used syncopoli to upload my phones photos over ssh to my home server nightly. I recently switched to iPhone and haven’t found a similar replacement yet.
It's not FOSS, but I've been pretty happy with Resilio Sync Home Pro. When I looked at file syncing software last (a few years ago), Syncthing was just not working well for me, although I can't remember the specific reason why -- I think it had something to do with Windows and Android support being weak at that time.
I keep a NAS set up as a master copy of all my shares and then use selective sync as appropriate on clients.
I would offer a quick recommendation against using Wasabi. They have an awful policy that forces you to pay for 90 days of storage, regardless if you store it for a single day or the full 90 days. I didn't realize this and have been paying them to not store my data. I switched to Backblaze B2 which doesn't have any shady business practices.
I ended up self-hosting everything, mostly because of the unnecessarily huge amounts of data I've got, combined with the power & flexibility of managing the infrastructure as desired. To keep things brief...
Eh, still on Outlook. I've got a ProtonMail account since the service's early days. Suffice to say, the worry that it won't be around is what made me hesitate to migrate to it. Clearly that hasn't happened, though it's been a few years since it came out, so what if it will happen sooner rather than later? ^^'
I could switch to POP3 for the Outlook emails so they're not stored on Microsoft's servers; however, that won't stop any copies of my emails from being stored on the servers, let alone in people's inboxes. With the latter in mind, using a privacy-oriented email service whilst your contacts use Outlook/Gmail/Yahoo! is counterproductive.
What I've used in the past:
it's been a few years, but I recall it to be very reliable and straightforward. Unfortunately, it's close-sourced, and I'm not a fan of Resilio Sync. Older versions of BitTorrent Sync still work, though they're limited in comparison to modern versions. Using an out of date, close-sourced software for dealing with data makes me raise an eyebrow for sure.Using it now, it's absolutely superb.
Syncthing: Excellent in the decentralised & privacy department, but initially unintuitive. Once the interface clicked, it was a joy using it. Unfortunately, things got murky when a lot of devices got connected, and things have been getting slow - the CPU usage was getting ridiculous when hashing tens of thousands of files. No iOS support, unfortunately.
Nextcloud: Very good web interface and decent syncing client. Mobile application was a massive plus. Deploying it wasn't too much of a hassle, and the mobile app was sweet. Sadly, I did find it to be slow and otherwise bloated for my needs. I find Nextcloud to be excellent in group/organisation settings, rather than solo usage.
What I ended up using:
2021: Resilio Sync. I gave Resilio Sync a whirl again, and despite its close-sourced nature, I'm impressed by the practical value it brings to the table. I've found it much more convenient and surprisingly lightweight than Syncthing. It's much leaner on resources, and features like the local-folder sync, ability to set predefined hosts rather than rely on trackers. The best one is the selective sync, which lets you selectively download files from a folder, instead of getting everything.
2019: Seafile. Pretty lightweight, open-source, straightforward web UI and an excellent syncing client (albeit with a few minor oddities). Some of the technical aspects are my cup of tea, such as block-based de-duplication and versioning of additions/changes/deletions.
I've ended up testing the platform with a few developers and designers around the world whom worked on the same project. Suffice to say, everything went very well overall. Conflicts were dealt with pretty easily, and any troubleshooting was easily resolved. Most of the issues were people's habits/usage patterns rather than technical quirks.
It also comes with a mobile syncing program. Not as great as the Nextcloud one; however, it does the job excellently in getting your files to and from the server. It has nifty treats like auto-uploading your gallery; hence, it's pretty good for backing up the gossip and Snapchat screenshots.
KeePassXC (or equivalent KeePass programs across different platforms). All synced through Seafile (and previously through the aforementioned syncing programs).
Bitwarden is very good, and technically makes cross-device usage much more painless than a KeePass database would.
2021: No bookmarks; instead, important articles are preserved locally. I've rarely returned to bookmarks after bookmarking them.
Plain Firefox Sync does the job for me. I contemplated hosting a Sync server a few times; however, I trust Mozilla with my data in that regard. Not only that, but they encrypt your data client-side before synchronising it to their servers.
Regarding tagging, Firefox does permit you to assign multiple tags to a bookmark. By the looks of it, you can store a bookmark in a folder in the sidebar, but also assign multiple/unrelated tags. ^^'
Eh, Outlook again. Bite me.
Programs and General Setup
Clonezilla is excellent for imaging an entire partition or disk. If you want to simply back up your configuration files, setting up a Git repository does the job.
You could go a step further and set the repository in your home folder, and have it ignore everything except the files you explicitly add to it.
If you're feeling particularly adventurous, you could set up the Git repository at the root level of your Linux installation, for the sake of being able to track configuration files outside of your home directory (e.g. in /etc). I haven't tried this one, nor do I condone it. ^^'
I used to store plain-text files in a Git repository for notes, diary entries, and everything... well... textual. ~~Recently, however, I've switched to Standard Notes. I'm finding the experience and sleekness to be a delight. Both the philosophy of security/longevity, and the technical aspects (data being encrypted, seamlessly backed up, minimalist base w/ fancy extensions), made the software worth getting a 5-year subscription for.
I would've switched earlier if it weren't for the abysmally painful process of getting self-hosting working. What ended up being the problem was poor/outdated documentation; however, I've written some notes on how to get the self-hosting working successfully if ever needed. Indeed, such notes are stored in Standard Notes. ^^'~~
Went back to Git repository with plain-text. It's as simple as it gets, with lots of flexibility and portability.
I started looking for alternatives to dropbox after my HDD failed on my laptop and - upon replacing it - realized that I wouldn't be able to link my new "device" to the account since I was probably grandfathered in to a device count well above the latest limit. I had a dream of setting up a home server and hosting my own files but I'm not very savvy with networking and would like to have access to them remotely without paying for a static IP address or some 3rd party service. After searching Tildes for threads similar to this one, someone mentioned Syncthing. I installed it on my arch linux laptop, windows 7 desktop, android phone, work computer. So far it has been incredibly convenient. Admittedly, I have no idea how it works. I'm assuming there must be some kind of centralized registration for device IDs or a fancy mesh network and blockchain. In any case, I had no trouble remotely connecting to my devices at home.
I also use KeepassXC and sync the database.
I use syncthing on all my devices and I find the experience mostly painless. I have a "main" instance on a VPS running 24/7, let's call it "the server". Every other device is connected only to the server, that way if a change happens on a device, the change is first transmitted to the server and then to the other devices once they get online. This setup is a good alternative to nextcloud.
The reason I say is mostly painless is because on mobile you need to manually add rules for every file or directory that you don't want to sync, but other that I'm very happy with it.
The only thing I keep in sync across devices is my KeePass database. I keep that in my Onedrive storage. Not open-source, but I imagine it's reliable. I use Keepass2Android on my phone and KeeWeb on my desktop. Though, I've thought about reverting to the native client.
I have a server under my desk running (among other things) Nextcloud which all my devices sync to. Disks are encrypted and extra sensitive stuff is gpg encrypted. I use the bookmarks plugin (plus rss and a few others); it also syncs contacts, calendar and photos on my phone. I'm very happy with it. Minimal sysadmin effort is required to keep everything running.
I also do manual offline backups using backintime. Nextcloud is not a backup; it is synchronisation.
I rely on a combination of Google Drive funded by my university, so it's Google minus the ads and tracking, and Bitorrent Sync (I think it goes by the name "Resilio Sync" these days). The first is great for data that I don't consider especially personal or mind a nosy university administrator seeing, like my personal music collection, or my homework / class resources. It works for everything that supports Google Drive, it's asynchronous, and comes with unlimited storage. I don't like using regular Google Drive, but I trust Google's legal department since it's an educational account, though I certainly understand if someone didn't.
For anything more personal though, bitorrent sync doesn't go through a permanent middle man, and it can get extremely high transfer speeds on a local connection, whereas Gdrive always goes through Google's servers. Downside is btsync doesn't work across the internet, at least the times I've tried it, and you need to have both devices on and connected to the internet at the same time.
I think you probably have some setting in resilio turned off that is making it not work across the internet. I have used resilio across the internet and it works perfectly. I think it has to phone home to it's own tracker to work, so if you turned off all the phone home settings, that probably caused it to not work.
I have used synching multiple times in the past. It has always been OK, but not good. This time when I was going to setup synching, I decided to try out resilio sync. I never thought to use it because I like Foss when I can use it and it isn't missing features or usability. Unfortunately synching imo can't hold a candle to resilio sync. I am using the free version. I have deployed a docker container, windows app, and android app. All have been completely painless and work 100%. If you like the idea of synching but are missing features or stability or ease of use, try out resilio sync.