I used to work in Data Recovery and have seen first hand, on countless occasions, just how devastating data loss can be, and how ludicrously expensive it can be to recover from... so you bet your...
Exemplary
Do you do backups? If so, how? (Cloud storage, offline backups, etc.)
I used to work in Data Recovery and have seen first hand, on countless occasions, just how devastating data loss can be, and how ludicrously expensive it can be to recover from... so you bet your ass I make backups! And I make sure to always abide by the 3-2-1 backup rule too:
Keep 3 backup copies of your data
Store 2 backups on different storage media (separate HDs is fine)
Store 1 backup copy offsite (encrypted on the cloud is fine)
And another important thing to keep in mind is that RAID is redundancynotbackup! I have seen way too many people get burned, by relying entirely on RAID to keep their data safe, to trust it for that. Failed array rebuilds are actually surprisingly common due to the stress that process puts on your remaining drives, and if all your drives are from the same manufacturing batch the likelihood they will fail at or near the same time increases even further. Failed arrays are also a massive PITA to recover data from, requiring highly specialized knowledge and software, and so it costs an absolute fortune to get done.
Also, don't manually back stuff up... you will inevitably forget at some point, or just get lazy, and will then pay the price for it. There are tons of great automated solutions available these days, both paid and free, so do some research, find one that suits your needs, and use it!
As for the rest of your questions: I move all OS/software created storage directories to somewhere other than my C drive (which is an NVMe and has limited space). I use a mix of commandline and file browsers, whichever will be faster or more convenient for the task. I rename most directories so they are listed in order of priority when sorted alphabetically. Media storage I have a 4TB NAS drive for, and use qBittorrent, iTunes, PLEX and calibre to keep everything organized. Note storage, I actually just use Notepad++'s cache for unimportant stuff, and save anything important to my Docs folder, which automatically gets backed up. Windows 10 is still my primary OS, but I have also been using WSL (Windows Subsystem for Linux) a lot more these days, as well as spinning up the occasional Linux VM.
I am waiting for the bad to happen (well, it actually already happened, i'm just dumb and continue doing the same shit), right now i have a total of 4TB in disks, two 1TB disks and one 2TB disk on...
RAID is redundancy not backup! I have seen way too many people get burned, by relying entirely on RAID to keep their data safe, to trust it for that
I am waiting for the bad to happen (well, it actually already happened, i'm just dumb and continue doing the same shit), right now i have a total of 4TB in disks, two 1TB disks and one 2TB disk on a really awkward setup, the two 1TB disks are in a RAID0 making a "md0" device of 2TB. Then i get the 2TB disk and the "md0" disk to make a raid1 array. In theory if either one of the disks fail it should continue working rock solid... but...
What happened was when i was upgrading from 1TB (those two 1TB disks in raid1) to 2TB (the current setup) i (or the software) messed something and i managed to somehow lose the data (deleted the partition table), recover it (got to recover the partition table with the backup ext4 superblock) and then lose it again, by writing on top of this disk.... Gezz i'm dumb...
I'm yet to start using borg, which seems like a good backup solution, but yeah, laziness is my enemy
That certainly is an "interesting" setup you have there. ;) I don't know if I would recommend it since you have basically doubled your failure potential for md0. At least your other drive in raid1...
That certainly is an "interesting" setup you have there. ;) I don't know if I would recommend it since you have basically doubled your failure potential for md0. At least your other drive in raid1 with it helps give you some actual redundancy, but I would still strongly recommend really trying hard to overcome your laziness to back up your essential files, at the very least. Even if you don't strictly abide by 3-2-1, just mirroring your important directories to another, non-raided drive will significantly reduce the chance of you losing your critical data.
then lose it again, by writing on top of this disk
LOL... oops. Yeah, that is rule #1 of Data Recovery; Never ever ever write to the drive you are trying to recover data from. This is why Data Recovery companies generally always attempt to image a drive first before doing anything else to it (unless there is risk of exacerbating physical damage to the drive by doing so), then continue their work using that image rather than the old drive, and finally only ever recover the data to another totally new drive.
By being RAID0? exactly, but somehow my head (or the software messed something) with the mount devices and paths and everthing got confused and this was the result :/ making a whole image of the...
basically doubled your failure potential for md0
By being RAID0?
Never ever ever write to the drive you are trying to recover
exactly, but somehow my head (or the software messed something) with the mount devices and paths and everthing got confused and this was the result :/ making a whole image of the disk wouldn't really be possible because i had no 2TB disk (i already had the data on the 2TB disk to "undo" the RAID0 that i had with the two 1 TB disks, then i would create the raid1 with those same 1TB disks put the data there and add the 2TB disk to the pool. Really confusing, guess it's understandable why i got everything confused :D)
Yup. RAID 0 has 0 redundancy, and so 2 drives in that config = twice the failure potential. If one of those drives fails, then md0 fails with it. Like I said though, at least you have some...
By being RAID0?
Yup. RAID 0 has 0 redundancy, and so 2 drives in that config = twice the failure potential. If one of those drives fails, then md0 fails with it. Like I said though, at least you have some redundancy thanks to the 2TB drive in RAID1 with md0... but, honest question for you; Why use RAID at all in this case? Why not just have all three of your drives separate and independent? Doing that will give you twice the space and it will also allow you to backup your more important data to multiple locations.
making a whole image of the disk wouldn't really be possible
Yeah, I realize that's not always possible to do for home users, and unless the data is worth more than the cost of the additional drives it would take to store the image, it isn't worth it anyways. But it is the only really surefire way to make sure you don't lose any additional data during a recovery.
That's actually a good question. For important data i could have a backup 3 times (in the 3 disks), I guess it's more a matter of being easier to organize the data just by having "one disk" But...
Why use RAID at all in this case?
That's actually a good question. For important data i could have a backup 3 times (in the 3 disks), I guess it's more a matter of being easier to organize the data just by having "one disk"
But then... with the current setup, how would i backup the data so i can "undo" the array? remove one of the disks (either the virtual md0 or the 2TB ) and then copy the data from the array to it? seems kinda risky. Getting an extra disk to handle the backups? This would be a nice option, but... $$$.
With me being as paranoid as I am in making sure I never lose any important data under any circumstances, I would probably just buy another drive (e.g. an external one, which is good to have for...
With me being as paranoid as I am in making sure I never lose any important data under any circumstances, I would probably just buy another drive (e.g. an external one, which is good to have for backup purposes anyways) and then copy everything to that first before I messed with the RAID config.
However, if that is not an option for you, do you have a friend with an external drive or a computer/laptop with enough HD space to temporarily hold your files? If so, you could just ask to borrow it for a few days, or have them come over to give you a hand.
Another option I just thought of... you could just pay for month of cloud storage (on a service like Backblaze. Onedrive or Dropbox), upload everything to that, fiddle with your array afterwards,...
Another option I just thought of... you could just pay for month of cloud storage (on a service like Backblaze. Onedrive or Dropbox), upload everything to that, fiddle with your array afterwards, and then download from there if something goes wrong.
hmm, that would be an option, but to be honest, i don't trust remote hosting services, ofc i could encrypt my data, but even then... Guess i would be better of getting a friend and asking him to...
hmm, that would be an option, but to be honest, i don't trust remote hosting services, ofc i could encrypt my data, but even then...
Guess i would be better of getting a friend and asking him to keep my data
You still want one copy off-site in case of fire, etc. One way to do it for an organization is to have a schedule for someone to bring encrypted hard drives back and forth so there is always one...
Exemplary
You still want one copy off-site in case of fire, etc. One way to do it for an organization is to have a schedule for someone to bring encrypted hard drives back and forth so there is always one off-site. Alternately it could be sent over an encrypted link to another site you control. Either way probably requires that there are periodic reminders to make sure that procedures are followed and the off-site backups actually work, as part of someone's job, and that job needs to get assigned to someone else when they leave.
For individuals I think cloud backup is more realistic. We all get busy and don't necessarily follow the backup procedures we decided on.
If you don't trust the cloud, even with encryption, but still need an offsite backup solution, you don't even need to take on the responsibility of setting up & maintaining that yourself either......
If you don't trust the cloud, even with encryption, but still need an offsite backup solution, you don't even need to take on the responsibility of setting up & maintaining that yourself either... there are a number of reputable offsite tape backup/vaulting services available to choose from that will do it all for you. @Loire
edit:
to make sure that procedures are followed and the off-site backups actually work
I just wanted to come back and emphasize how crucial that part really is. It honestly wasn't all that rare to see organizations that thought they had good backup policies get totally screwed when, after catastrophic on-site data loss, they then found out that their offsite backup drives didn't actually work... and/or worse still, didn't contain the critical data that they thought they did.
As slow and archaic as they may seem, automatic tape backup systems still exist for good reasons; they are far more stable and reliable, often with far higher capacity, than most platter drives. When it comes to comprehensive, large-scale, long-term data backup, magnetic tapes are still king, IMO.
See also: this thread Important PSA I lost approximately 5 years of data in my first hard drive crash, before discussion of healthy backup habits was commonplace. It was devastating. That event...
I lost approximately 5 years of data in my first hard drive crash, before discussion of healthy backup habits was commonplace. It was devastating. That event taught me the importance of backups firsthand, and I have paid for the piece of mind that comes with automated cloud backup ever since.
I have since also had to do full recoveries from cloud backups twice due to sudden hard drive failures. Had I not had those in place, I would have lost even more than I did the first time. I can't recommend a specific company for this, as both of the ones I've used have stopped operating (Mozy was bought out and sunsetted, and CrashPlan stopped its consumer plans and now focuses only on business clients). All I can say is that any company is better than none.
Please, please, PLEASE back up your data! Also, do NOT do it manually! You will forget and you will lose important stuff. Drive failures never happen at a convenient or predictable time.
With that said, currently for backups I use Tresorit which is a privacy-friendly cloud backup/storage that's very similar to Dropbox in terms of features and execution. Normally the pricing would be too costly for it to be worth it, but I have it on a pretty steep discount at the moment. If they ever start charging me full price, I'll probably hop to a different service.
Tresorit savings tip
Sign up and use their free-trial. At the end of the trial, terminate and give your reason for doing so as the prohibitive cost. They might offer you a much better rate, which is what they did for me. This tip goes for more than just Tresorit, by the way. I've gotten discounts for other services by this same method.
I've never had to do a full recovery from Tresorit, which is the true test of a backup service, but dead simple cloud backup options are limited on Linux, and Tresorit is more feature rich than SpiderOak.
I am excited for the day that NextCloud has options that are both dead simple and affordable. I'll use that the moment it fits that bill, as I feel like it's the best of the options out there from a privacy standpoint. I also miss Google Drive's online document editing, so being able to use Collabora would be great.
As for actual organization, I've gone on so many different organization and restructuring kicks and none have really fully panned out. Instead, I've organized portions of my stuff so that certain sections are okay, but it doesn't really follow a proper taxonomy, structure, or hierarchy. Instead everything is just all there, in chunks, and I approximately know where everything is. One of the downsides of trying to organize is that doing so messes up my mental map, so it's often more difficult to find stuff after I rearrange things, simply because they're not where they used to be. This makes organization feel counterproductive even when it most certainly isn't.
I sometimes still have dreams of one day getting everything in its place and getting rid of the cruft I've accumulated over the decades, but I don't know that it'll ever actually happen. I've found the most productive thing I can do is just make peace with my disorganization and know that it's okay if my files are strewn about in a way that's less than ideal. One of the virtues of digital spaces over physical ones is that disorganization doesn't really take up room or ruin aesthetics. It can be easily ignored or cordoned off. As such, I don't know that chasing perfection is worth the hassle, especially when accepting things as they are comes with nearly no downsides.
I have my more important personal documents in Google Drive and notes in Google Keep. I also do occasional backups of my desktop computer to an external hard drive using Time Machine. I push git...
I have my more important personal documents in Google Drive and notes in Google Keep. I also do occasional backups of my desktop computer to an external hard drive using Time Machine. I push git repos for programming projects to GitHub if they're public or sync them to my laptop using Keybase if they're private.
I sometimes have project folders organized by language, for example putting Dart projects under a "dart" directory. I've also tried putting my more formal projects under a "projects" directory and less formal ones in an "experiments" directory, but find that organizing by language makes it easier to remember where single-language projects are after not working on them for a while.
Most IDE's will suggest a default place for project folders but I override it.
I used to over-engineer this stuff. I had custom partitions, organized my software into D:\Programs and D:\Games, ran extensive backups, etc. I don't do that anymore. After I started dualbooting...
I used to over-engineer this stuff. I had custom partitions, organized my software into D:\Programs and D:\Games, ran extensive backups, etc.
I don't do that anymore. After I started dualbooting Linux, I began focusing on portability over complex setups. Now I store important files in Dropbox (documents, projects, whatever else), which syncs nicely between OSes and offers a layer of redundancy.
I dumped my extensive music library on Google Play Music, but Apple's locker works fine too. I'm not a photographer but I'd use Google Photos or similar there too. I stopped hording movies which I'd have never watched and just use Netflix/Plex/whatever instead.
What I've found is that it's a whole lot easier to backup the dozens of files that really matter than TBs of data. My Dropbox is less than 1.5GB full now. Programming projects, legal documents; even a section for Data like key-signing certificates, game saves, and dotfiles.
Basically I embraced digital minimalism. Instead of building a complex and perfectly-curated system, I've started treating it as a potentially-volatile space. It could crash at any time, and that's okay. At worst I'll have to reinstall some software, and without worrying about install paths or symlinks that takes significantly less time now.
I guess this has become part 3 of my "minimize everything digital" series [1][2].
I self-host Nextcloud on a server at home, which backs up to a pi with an external HDD off-site. I manage these backups with Borg. For all of the things not in Nextcloud (config files, etc.), I...
Do you do backups? If so, how? (Cloud storage, offline backups, etc.)
I self-host Nextcloud on a server at home, which backs up to a pi with an external HDD off-site. I manage these backups with Borg. For all of the things not in Nextcloud (config files, etc.), I also use Borg to backup to the same Pi. I just recently had to rebuild my entire stack, so I am confident that these backups work, although I have not sufficiently automated restoring from them.
Do you use your OS's default user folders (Documents, Pictures, Downloads, Desktop, etc.) or do you opt for a more custom scheme?
I have used my system defaults with a few extras for a while, but am just now experimenting with something more custom. At the very least, I renamed all files to be lowercase to make command line navigation easier. I also removed some directories I don't use, and added a few top level that I felt were helpful (I will cover these in answers below).
Do you use your browser's default download directory, or do you do something more custom?
I have ~/download set to be my default, but I have my browser ask me where to download every time. It's annoying 30% of the time, and useful 70% of the time.
Do you navigate through your files using a file browser, the command line, or both?
Both, really. It depends on the file and what I need to do with it.
Do you have any naming conventions for files and folders? (Dash/underscore/space delimiting? Lower/uppercase?)
Mostly camel case, but the files are really not as standardized as they should be.
Where do you store code repositories (if that's something you do), and do you make a distinction between code repositories and documents?
I have ~/code/ in my home folder for code repositories. Since they are almost all git repositories that are backed up elsewhere, they are not usually included in my backup regime.
How do you deal with programs that automatically create directories in your workspaces without your permission? (I'm looking at you, IDEs with "Projects" folders.)
I don't have a good answer to this yet. One I have heard, but not tried, is setting your home directory to be read-only, but I don't know how much that may or may not break things.
How do you handle media storage? Do you use any library managers for ebook/pdf files, image files, music files, video files, etc.?
I use Calibre for e-books, Airsonic (soon to be moved to Funkwhale) on my server for music, and Jellyfin on my server for videos. All images and documents are in Nextcloud, and I manage those mostly by hand.
How do you handle document/note storage? Do you use files in the Documents folder, or do you use an alternative method like cloud-based office tools, or some sort of wiki-ish program like Indigrid or Trillium?
Mostly I use a file hierarchy in Nextcloud, although not all of it is well organized. My school folder is well organized since the input data is nice and hierarchical and I use it regularly. The hierarchy goes: School name >> Semester >> Class >> Project. My photos are partially organized. The old ones from before my organization are a mess, but any recent ones are synced into a folder based on the device they come from, and they retain timestamps. My documents are a mess, I have some folders to help, but nothing cohesive.
If you use a Linux-based OS, where do you put sandboxed/portable applications (e.g. snap, flatpak, appimage)?
I also have a ~/programs/ directory in my home folder where I mostly keep these. Really, this folder is for code I do not intend to hack on, just to use, so there are some compiled git repositories in there as well. ~/code/ is exclusively for source code I want to look at or manipulate, and is mostly git repos and some extracted tar files.
I have a folder called !!! Data at the top of my profile folder. I keep everything of value in there, in case I find myself having to switch platforms again. (If I can afford to, I would also...
I have a folder called !!! Data at the top of my profile folder. I keep everything of value in there, in case I find myself having to switch platforms again. (If I can afford to, I would also transfer whatever savegame files I have lying around. If not, at least I have everything else stored.)
!!! Data hosts a handful of folders – mostly categories by which I handle the files – as well as !!! Inbox, again at the very top.
Everything that I don't use immediately, I put into the inbox folder. It can be cluttery at times because it's allowed to be: it's a kitchen sink of all the files that are between destinations. The inbox folder has a few subfolders of its own, mostly for reduction of visual clutter, but also for cleaner transition between itself and the destination folder. Images to go !!! Inbox\@ images, videos go to !!! Inbox\@ videos, text files of varied kinds go to !!! Inbox\books and such etc.
I have a vague convention on naming folders: if you want something at the top (perhaps because it's a parent category of its own), you prepend exclamation marks and a single whitespace to its name. If it's a subcategory (like @ images is for !!! Inbox), it gets at-signs and a single whitespace in front. There's also the # octothorpe categorization, which is roughly for folders that are important in themselves (like !!! Data\# Writing, which stores all of my tales and some of the notes). None of it has been formally defined: it's something I follow on an intuitive level, retooling the system as necessary.
Which, frankly, describes much of my workflow. If something needs a separate category because it's gaining traction as a fraction of the contents, it gets a category. I have !!! Data\Markdown Documents because I have enough of those. I have !!! Data\Text because I make a lot of plain-text notes that fit nowhere else.
If I need to write something down now, I use a plain-text file of vague, ever-evolving formatting, by calling a set shortcut to the Notepad app. I then save it to the desktop: it serves as an inbox for notes, of sorts. That way, I can later access the notes in an instance, without having to trawl through a hundred files.
Which is to say: the inbox is there because I never categorize immediately. It may take a month before I even touch the folder again. Until then, the files are there, and I'm safe in the knowledge that eventually, I'll get around to it, because I always do.
Frankly, the only reason I don't use Indigrid for the immediate notes is because it doesn't have that feature. I've spoken to Mark about it, and he's given me no signal of interest. (To be fair, he never does: he probably just jots it down and implements it down the road, like he did with my virtual-space suggestion.) It's something I'd like to do with Intergrid, now that I also have the idea of making it into a new-tab browser extension.
Between the plain-text notes and the Intergrid intervention, I found myself using a different browser extension – My Notes, which turns the new tab into a plain-text note-taker – rather comfortably.
As a sidenote, I also store the installers for apps I find myself using with some regularity in a separate category within the data folder. I had, at one point, found myself having to transition between fresh installments of an OS due to failure in HDDs and SSDs, so I figured: having the batch of apps ready without having to find them later on would prove beneficial.
As for backups... I am woefully behind times on this. I save it all maybe twice a year, which I recognize is egregiously-insufficient, especially considering the trouble I went through to consolidate all of my valuable digital resources. I'm looking for a solution that, once set up, will require minimal action on my side to perform. I have an external HDD I use for these backups, and I'd prefer to have at least a local-first solution, and ideally one that backs-up both to the HDD and to a reliable cloud server.
I’ve set up my life such that anything of consequence is within a cloud service - I use Dropbox heavily and Google photos for pictures and videos I make, Google drive for most documents, Evernote...
I’ve set up my life such that anything of consequence is within a cloud service - I use Dropbox heavily and Google photos for pictures and videos I make, Google drive for most documents, Evernote for notes, source code on github, etc. I don’t have significant media now that streaming is so common, so I got rid of a lot of my local storage needs.
This means that I have no need to “back up” anything, and it is already accessible via mobile as well. This also means that setting up a new phone is super easy, there’s essentially no data I need to worry about copying over. The only challenge here has been text messages, since I don’t trust iCloud as it seems crappy.
A word of caution: I recently reclaimed all my data from Google after relying on it for storage and backup of my files and photos for years. And it sucked. Google Takeout will create archives for...
A word of caution: I recently reclaimed all my data from Google after relying on it for storage and backup of my files and photos for years. And it sucked. Google Takeout will create archives for download, but it takes forever and is not necessarily organized nicely once you decompress them. If you have a lot of photos, it is very cumbersome.
Also, as someone else mentioned, it’s good to have physical backups as well ;) I have unlimited storage on Google through my uni, but for lots of reasons I’ve been working toward self-hosting Nextcloud and other similar services instead. I highly recommend it!
Yeah, it’s a point well taken. For a while I was actually double syncing to both google photos and dropbox for redundancy. Maybe I should do that again.
Yeah, it’s a point well taken. For a while I was actually double syncing to both google photos and dropbox for redundancy. Maybe I should do that again.
I have been tinkering on my backup system for months now. Right now, I am using Backblaze B2 and rsync; its dirt cheap and works. But the data isn't encrypted at rest and it is fairly fragile. So,...
I have been tinkering on my backup system for months now. Right now, I am using Backblaze B2 and rsync; its dirt cheap and works. But the data isn't encrypted at rest and it is fairly fragile. So, I am slowly switching to ZFS as the filesystem and trying out snapshots. I am also looking at rsync.net for the storage, as I do love me some open-source tools. I am also looking into git-annex for a more versatile software component.
Using zfs as your filesystem will also add another protection layer to help prevent bit-rot through the zfs checksums. This is obviously not a backup strategy, but once you setup a scheduled zfs...
Using zfs as your filesystem will also add another protection layer to help prevent bit-rot through the zfs checksums. This is obviously not a backup strategy, but once you setup a scheduled zfs scrub (I scheduling mine via cron once a month) you can help prevent unnoticed bit-rot from happening that you may not even know when it happened to restore from a backup.
Newer zfs version 0.8.x offer native encryption if this is something you want to explore. You can also use zfs replication of the encrypted filesystem to a remote system and leave it encrypted such that the remote system cannot read the data. Basically you can keep a remote "untrusted" system up to date with your backups and not leave the key on that system.
Backups: external USB drives stored at my house, at my work, at the bank. Storage: One folder with all my stuff less sensitive stuff grouped into subfolders called Photos, Music, HTML etc......
Backups: external USB drives stored at my house, at my work, at the bank.
Storage: One folder with all my stuff less sensitive stuff grouped into subfolders called Photos, Music, HTML etc... Sensitive data in an encrypted image.
I use the file browser to move stuff into my one folder/ encrypted image.
Notes are in a word document or in the native notes app.
I dont have sandboxed apps, but now I am thinking I should....
I built my own sever at home which serves as one location for backups and important documents. I still keep documents, like for school, on my computer of course since I can't always just be...
I built my own sever at home which serves as one location for backups and important documents. I still keep documents, like for school, on my computer of course since I can't always just be connected to my server. I also backup to the cloud with Backblaze. Lastly, I have a physical backup drive where I use Carbon Copy Cloner as I use a Mac.
On my computer (which is a laptop), I organize everything by folder and relevance. So there is one large folder for "school" which then has sub-folders for different grades which have sub folders for subjects, etc. I also use alfred on my mac which allows me to navigate and search for a folder or file very easily.
I am thinking of setting up syncthing on my mac with my server to make it a little easier. Or I could do it through nextcloud. Not sure yet, and if anyone has any experience with a setup like this, let me know!
If you have any more specific questions, let me know. Happy to answer them!
I've gone through numerous iterations of data re-organizations over the years, but lately I'm going through a significant digital minimalism phase. While i still want to preserve milestone and...
I've gone through numerous iterations of data re-organizations over the years, but lately I'm going through a significant digital minimalism phase. While i still want to preserve milestone and keepsake stuff, I'm getting to a point where I think more and more stuff is ephemeral and really need not be archived/saved/preserved.
That being said, i have always found it valuable when i learn of how others manage their data, so here goes my setup...
I'm not very great at backups. I use dropbox and onedrive, but back up the data from both of them to a server in my house using samba. This obviously should be upgraded to a more automated style of the 3-2-1 method of backing data up...so that my future project right after new years. I do like the fact that both dropbox and onedrive have a "revert" function where you can turn the clock back a day or some to "go back in time" in case you totally kill some essential files. Is it a great back up strategy? Of course not! but i do like it, and have used it once.
BTW, currently I use dropbox for my most essential - and core - files. This comes to about 1.75 GB, so its not a big deal. I use MS OneDrive for my family because it comes with the MS Office 365 subscription, and the storage is quite generous...So i store my family's photos on it. Functionally speaking, i have not seen a difference between onedrive and dropbox...And the ONLY reason i even use dropbox, is because there is no official onedrive client for linux...and linux is my daily driver OS. But if/whenever microsoft creates a native linux client, because of lower cost, i might just move everything to onedrive. I do self-host my own nextcloud server...but i've not run sufficient tests to ensure preventing data loss...also, my nextcloud server is hosted on a cheap vps, but storage there is not as cheap (at least for the amount of data that i'm saving.) Nextcloud would be my preferred storage for local and remote file storage and synching/sharing...but i'm not there just yet. for now, its a combo of dropbox and onedrive.
On linux i do place everything under my /home/$USER directory hierachy, but since i use dropbox, it is really all under /home/$USER/DropBox/...etc... At all my dayjobs, i unfortunately get issued a windows machine, so just have everything under whatever DropBox folder gets set up by the dropbox app - i think thats under the typical Windows OS user dir. hierarchy, but i don't exaclty recall, nor care since its all under dropbox.
I do use whatever default download folder my OS or browser set. The stuff there is always considered ephemeral, so i care nothing about it after i move the files away from it.
I navigate both from the file browser and command line - it really depends on what I'm doing. But as i noted above about going through digital minimalism, i find it refrreshing navigating through command line more and more. Though this isn't possible or convenient with all desktop apps. Again, it depends.
I name almost all of my files and folders lowercase and almost never with spaces. I prefer hyphens/dashes to separate the terms used in a filename/folder name - e.g. some-file-name.txt, some-folder-name, etc. I find this easier for scripting, etc. Also, i avoid using special characters within the filenames for fear the Window OS side might mess up a file or something. Also, whenever needing to use dates in a file/folder name, I always use the iso format of YYYY-MM-DD, and that solves my sorting needs quite easily. If - as others have noted - i wish to somewhat permantently sort some files (to have them hsow up at the top, etc.), then I'll prefix file names with numbers. One file that almost always exists in my folders is the 1-README.md (or sometimes 01-README.md) file...This is a convention i use to help explain "why" i have the current directory in existence. Part of it is to help with my archival reasoning, another part is WHEN i die, it will help my family know what stuff is contained in the current dir. With my digital minimalism i find les use for the 1-README file, since I've recently greatly reduced complexity of my data org....But I'm hoping this file will still help my future family.
I'll add that as far as my data org., here's what i've been using lately, though might reduce this even further in the coming days:
[whatever user directory]/dropbox/
1-drafts/ (Folder of active note/txt files. At the end of each year i move them elsewhere - see below.)
some-meeting-2019-12-20.md
some-other-random-text-note.md
other notes files
1-now/ (This is like my inbox - basically what I'm working on "now", where i place all manner of uncategorized files...but have used this so rarely now that will likely remove it soon.)
apps/ (This is a hierarchy of apps, scripts, and environments for local apps, etc.)
finances/ (Um, a hierarchy of financial related files.)
hr/ (This is my resume, portfolio for getting jobs, a spreadsheet for tracking jobs applied for, downloaded job descriptions, and any other stuff related to getting a new job.)
library/ (This is sort of a catch-all reference library where i save material that is passive that i might refer to later...think of it as my own, hand-crafted manual pages, as well as reference PDF or HTML files from others.)
logs/ (In this hierarchy, at the end of each year, i "archive" the notes from the "1-drafts" folder to here under a year subfolder - e.g. 2018/, 2019/, etc..)
2019/
2018/
...etc.../
media/ (Um, a hierarchy of media files including images like logos/icons, photos, videos, etc.)
ops/ (Ah, this is my projects hierarchy. Tons of stuff that i'm currently working on is held here. This includes coding as well as non-coding projects, even home DIY projects, i mean everything that could be considered a project is saved here. All assets related to its relevant project are held within a single, relevant folder under this "ops" folder hierarchy. i use the term "ops" as an equivalent to "operations" because for some reason its faster for me to type ops than, say, projects, etc. Weird, i know, but it works for me.)
project-abc/
project-123/
...etc.../
I hope my comments help. I'd be happy to answer any other questions you might have!
I do a local time machine backup and have Arq backing up my data to Backblaze b2. My documents are all in my ~/Documents folder synced to a VPS using Syncthing so I can access the files on the go...
I do a local time machine backup and have Arq backing up my data to Backblaze b2. My documents are all in my ~/Documents folder synced to a VPS using Syncthing so I can access the files on the go as well.
Code and notes are stored in public and private GitHub repos, I do this for notes mostly so I can have them available on the go as well with WorkingCopy on iOS.
I edit the files on my mac with nvUltra (beta). I switched from NValt to nvUltra and NValt wasn't capable of having a folder hierarchy so I prefix my notes with somewhat of a tag or category. But I mostly use the search functionality.
I actually like my setup but somehow I'm tempted to switch to a hosted solution for my notes like google docs, notion, onenote or office 365. I don't want to take care of the management of things like Syncthing to keep things in sync but I also don't want to give up my files to a weird format like gdocs.
I use Google Drive for as much free cloud storage as I can get, and keep the other stuff backed up onto a separate drive. I don't use any automatic cloud backup programs, because I don't know of...
I use Google Drive for as much free cloud storage as I can get, and keep the other stuff backed up onto a separate drive. I don't use any automatic cloud backup programs, because I don't know of any good ones. Anybody got a cheap recommendido?
I, uh, really should have backups. But I don't really do anything as much as I should. Right now, my current setup includes a basic SATA SDD which I use as a boot drive plus applications. I...
I, uh, really should have backups. But I don't really do anything as much as I should.
Right now, my current setup includes a basic SATA SDD which I use as a boot drive plus applications. I combine that with two HDDs; one is used specifically for video games (they just take up too much space!) and the other is used for my actual important data. Those are in a really simple hierarchy based on what they actually are.
The good thing about separating my data drive from the OS and applications is that it means that it gets significantly less access and wear, so it will last longer. And as it grows, I put it on larger and larger drives, so they are getting replaced over time. So even if I am not doing backups like I should be, I am still doing proactive data maintenance to prevent catastrophic failure.
I do use the default download location for most things, but that's because they're often ephemeral. I'll go through them every now and again and delete them.
I plan to build a NAS sometime in the future, but honestly I don't have the money to buy a good solution (like a Synology system) or time to implement a cheaper version.
The 3 backups rule is definitely the normal advice. One local copy, another local copy on a different machine or media, and another copy offsite. I'm doing OK by this standard, but I've got some...
The 3 backups rule is definitely the normal advice. One local copy, another local copy on a different machine or media, and another copy offsite. I'm doing OK by this standard, but I've got some work to do. I bought two 10TB EasyStore HDDs from a Best Buy sale about a year ago, shucked one and put it in my PC, and kept the other in its external enclosure connected to my PC. I have Veeam for Windows backing up the internal HDD to the external HDD every night automatically. Otherwise, I have files strewn a bit across OneDrive (have 1TB as part of O365), DropBox, iCloud, and other PCs. I really don't have enough cloud space to truly follow the rule of 3, since my local data is ~2TB, so I need to figure that out shortly. Maybe BackBlaze?
I backup important things on my second PC and external storage devices and add more redundancy with rare things that absolutely need to keep their backups. I have very particular and practical...
I backup important things on my second PC and external storage devices and add more redundancy with rare things that absolutely need to keep their backups.
I have very particular and practical naming conventions for music and media, but don't worry too much about anything else as it will be organized into an appropriately named folder and easy to find. Which I use a program called "Everything" that lets you type and search in real time, very handy tool for finding things on your PC.
For folders I avoid the OS provided folders as they generally have shitty preset parameters on them compared to a new and independently named folder. I also like to keep my storage drives separate from the OS drive, first so that they don't all crash at once and corrupt a bunch of files and second so they are easily swapped into another PC should I have other hardware problems. Also less shit on the SSD should prolong its lifespan a little more.
I use MediaMonkey for a music player, MPC-HC for a video player. ninite.com has a good selection of free programs that you can choose from.
I don't backup anything now. Only the most important files (encrypted files with passwords). I use custom directories. ~/data is my HDD with the bulk of what i have ~/data/personal with personal...
I don't backup anything now. Only the most important files (encrypted files with passwords).
I use custom directories.
~/data is my HDD with the bulk of what i have
~/data/personal with personal files like paychecks, journal, etc
~/data/media/{music,videos,pictures,books,other}
~/download/{torrent,soulseek}
~/bin with my custom programs and scripts
~/dev for programming projects
Everything is lowercase. My music is organized in ~/data/media/music/artist/year - album folders.
I used to work in Data Recovery and have seen first hand, on countless occasions, just how devastating data loss can be, and how ludicrously expensive it can be to recover from... so you bet your ass I make backups! And I make sure to always abide by the 3-2-1 backup rule too:
And another important thing to keep in mind is that RAID is redundancy not backup! I have seen way too many people get burned, by relying entirely on RAID to keep their data safe, to trust it for that. Failed array rebuilds are actually surprisingly common due to the stress that process puts on your remaining drives, and if all your drives are from the same manufacturing batch the likelihood they will fail at or near the same time increases even further. Failed arrays are also a massive PITA to recover data from, requiring highly specialized knowledge and software, and so it costs an absolute fortune to get done.
Also, don't manually back stuff up... you will inevitably forget at some point, or just get lazy, and will then pay the price for it. There are tons of great automated solutions available these days, both paid and free, so do some research, find one that suits your needs, and use it!
As for the rest of your questions: I move all OS/software created storage directories to somewhere other than my C drive (which is an NVMe and has limited space). I use a mix of commandline and file browsers, whichever will be faster or more convenient for the task. I rename most directories so they are listed in order of priority when sorted alphabetically. Media storage I have a 4TB NAS drive for, and use qBittorrent, iTunes, PLEX and calibre to keep everything organized. Note storage, I actually just use Notepad++'s cache for unimportant stuff, and save anything important to my Docs folder, which automatically gets backed up. Windows 10 is still my primary OS, but I have also been using WSL (Windows Subsystem for Linux) a lot more these days, as well as spinning up the occasional Linux VM.
I am waiting for the bad to happen (well, it actually already happened, i'm just dumb and continue doing the same shit), right now i have a total of 4TB in disks, two 1TB disks and one 2TB disk on a really awkward setup, the two 1TB disks are in a RAID0 making a "md0" device of 2TB. Then i get the 2TB disk and the "md0" disk to make a raid1 array. In theory if either one of the disks fail it should continue working rock solid... but...
What happened was when i was upgrading from 1TB (those two 1TB disks in raid1) to 2TB (the current setup) i (or the software) messed something and i managed to somehow lose the data (deleted the partition table), recover it (got to recover the partition table with the backup ext4 superblock) and then lose it again, by writing on top of this disk.... Gezz i'm dumb...
I'm yet to start using borg, which seems like a good backup solution, but yeah, laziness is my enemy
That certainly is an "interesting" setup you have there. ;) I don't know if I would recommend it since you have basically doubled your failure potential for md0. At least your other drive in raid1 with it helps give you some actual redundancy, but I would still strongly recommend really trying hard to overcome your laziness to back up your essential files, at the very least. Even if you don't strictly abide by 3-2-1, just mirroring your important directories to another, non-raided drive will significantly reduce the chance of you losing your critical data.
LOL... oops. Yeah, that is rule #1 of Data Recovery; Never ever ever write to the drive you are trying to recover data from. This is why Data Recovery companies generally always attempt to image a drive first before doing anything else to it (unless there is risk of exacerbating physical damage to the drive by doing so), then continue their work using that image rather than the old drive, and finally only ever recover the data to another totally new drive.
By being RAID0?
exactly, but somehow my head (or the software messed something) with the mount devices and paths and everthing got confused and this was the result :/ making a whole image of the disk wouldn't really be possible because i had no 2TB disk (i already had the data on the 2TB disk to "undo" the RAID0 that i had with the two 1 TB disks, then i would create the raid1 with those same 1TB disks put the data there and add the 2TB disk to the pool. Really confusing, guess it's understandable why i got everything confused :D)
Yup. RAID 0 has 0 redundancy, and so 2 drives in that config = twice the failure potential. If one of those drives fails, then md0 fails with it. Like I said though, at least you have some redundancy thanks to the 2TB drive in RAID1 with md0... but, honest question for you; Why use RAID at all in this case? Why not just have all three of your drives separate and independent? Doing that will give you twice the space and it will also allow you to backup your more important data to multiple locations.
Yeah, I realize that's not always possible to do for home users, and unless the data is worth more than the cost of the additional drives it would take to store the image, it isn't worth it anyways. But it is the only really surefire way to make sure you don't lose any additional data during a recovery.
That's actually a good question. For important data i could have a backup 3 times (in the 3 disks), I guess it's more a matter of being easier to organize the data just by having "one disk"
But then... with the current setup, how would i backup the data so i can "undo" the array? remove one of the disks (either the virtual md0 or the 2TB ) and then copy the data from the array to it? seems kinda risky. Getting an extra disk to handle the backups? This would be a nice option, but... $$$.
What would you do in this scenario?
With me being as paranoid as I am in making sure I never lose any important data under any circumstances, I would probably just buy another drive (e.g. an external one, which is good to have for backup purposes anyways) and then copy everything to that first before I messed with the RAID config.
However, if that is not an option for you, do you have a friend with an external drive or a computer/laptop with enough HD space to temporarily hold your files? If so, you could just ask to borrow it for a few days, or have them come over to give you a hand.
Another option I just thought of... you could just pay for month of cloud storage (on a service like Backblaze. Onedrive or Dropbox), upload everything to that, fiddle with your array afterwards, and then download from there if something goes wrong.
hmm, that would be an option, but to be honest, i don't trust remote hosting services, ofc i could encrypt my data, but even then...
Guess i would be better of getting a friend and asking him to keep my data
You still want one copy off-site in case of fire, etc. One way to do it for an organization is to have a schedule for someone to bring encrypted hard drives back and forth so there is always one off-site. Alternately it could be sent over an encrypted link to another site you control. Either way probably requires that there are periodic reminders to make sure that procedures are followed and the off-site backups actually work, as part of someone's job, and that job needs to get assigned to someone else when they leave.
For individuals I think cloud backup is more realistic. We all get busy and don't necessarily follow the backup procedures we decided on.
If you don't trust the cloud, even with encryption, but still need an offsite backup solution, you don't even need to take on the responsibility of setting up & maintaining that yourself either... there are a number of reputable offsite tape backup/vaulting services available to choose from that will do it all for you. @Loire
edit:
I just wanted to come back and emphasize how crucial that part really is. It honestly wasn't all that rare to see organizations that thought they had good backup policies get totally screwed when, after catastrophic on-site data loss, they then found out that their offsite backup drives didn't actually work... and/or worse still, didn't contain the critical data that they thought they did.
As slow and archaic as they may seem, automatic tape backup systems still exist for good reasons; they are far more stable and reliable, often with far higher capacity, than most platter drives. When it comes to comprehensive, large-scale, long-term data backup, magnetic tapes are still king, IMO.
See also: this thread
Important PSA
I lost approximately 5 years of data in my first hard drive crash, before discussion of healthy backup habits was commonplace. It was devastating. That event taught me the importance of backups firsthand, and I have paid for the piece of mind that comes with automated cloud backup ever since.
I have since also had to do full recoveries from cloud backups twice due to sudden hard drive failures. Had I not had those in place, I would have lost even more than I did the first time. I can't recommend a specific company for this, as both of the ones I've used have stopped operating (Mozy was bought out and sunsetted, and CrashPlan stopped its consumer plans and now focuses only on business clients). All I can say is that any company is better than none.
Please, please, PLEASE back up your data! Also, do NOT do it manually! You will forget and you will lose important stuff. Drive failures never happen at a convenient or predictable time.
With that said, currently for backups I use Tresorit which is a privacy-friendly cloud backup/storage that's very similar to Dropbox in terms of features and execution. Normally the pricing would be too costly for it to be worth it, but I have it on a pretty steep discount at the moment. If they ever start charging me full price, I'll probably hop to a different service.
Tresorit savings tip
Sign up and use their free-trial. At the end of the trial, terminate and give your reason for doing so as the prohibitive cost. They might offer you a much better rate, which is what they did for me. This tip goes for more than just Tresorit, by the way. I've gotten discounts for other services by this same method.I've never had to do a full recovery from Tresorit, which is the true test of a backup service, but dead simple cloud backup options are limited on Linux, and Tresorit is more feature rich than SpiderOak.
I am excited for the day that NextCloud has options that are both dead simple and affordable. I'll use that the moment it fits that bill, as I feel like it's the best of the options out there from a privacy standpoint. I also miss Google Drive's online document editing, so being able to use Collabora would be great.
As for actual organization, I've gone on so many different organization and restructuring kicks and none have really fully panned out. Instead, I've organized portions of my stuff so that certain sections are okay, but it doesn't really follow a proper taxonomy, structure, or hierarchy. Instead everything is just all there, in chunks, and I approximately know where everything is. One of the downsides of trying to organize is that doing so messes up my mental map, so it's often more difficult to find stuff after I rearrange things, simply because they're not where they used to be. This makes organization feel counterproductive even when it most certainly isn't.
I sometimes still have dreams of one day getting everything in its place and getting rid of the cruft I've accumulated over the decades, but I don't know that it'll ever actually happen. I've found the most productive thing I can do is just make peace with my disorganization and know that it's okay if my files are strewn about in a way that's less than ideal. One of the virtues of digital spaces over physical ones is that disorganization doesn't really take up room or ruin aesthetics. It can be easily ignored or cordoned off. As such, I don't know that chasing perfection is worth the hassle, especially when accepting things as they are comes with nearly no downsides.
I have my more important personal documents in Google Drive and notes in Google Keep. I also do occasional backups of my desktop computer to an external hard drive using Time Machine. I push git repos for programming projects to GitHub if they're public or sync them to my laptop using Keybase if they're private.
I sometimes have project folders organized by language, for example putting Dart projects under a "dart" directory. I've also tried putting my more formal projects under a "projects" directory and less formal ones in an "experiments" directory, but find that organizing by language makes it easier to remember where single-language projects are after not working on them for a while.
Most IDE's will suggest a default place for project folders but I override it.
I used to over-engineer this stuff. I had custom partitions, organized my software into
D:\ProgramsandD:\Games, ran extensive backups, etc.I don't do that anymore. After I started dualbooting Linux, I began focusing on portability over complex setups. Now I store important files in Dropbox (documents, projects, whatever else), which syncs nicely between OSes and offers a layer of redundancy.
I dumped my extensive music library on Google Play Music, but Apple's locker works fine too. I'm not a photographer but I'd use Google Photos or similar there too. I stopped hording movies which I'd have never watched and just use Netflix/Plex/whatever instead.
What I've found is that it's a whole lot easier to backup the dozens of files that really matter than TBs of data. My Dropbox is less than 1.5GB full now. Programming projects, legal documents; even a section for Data like key-signing certificates, game saves, and dotfiles.
Basically I embraced digital minimalism. Instead of building a complex and perfectly-curated system, I've started treating it as a potentially-volatile space. It could crash at any time, and that's okay. At worst I'll have to reinstall some software, and without worrying about install paths or symlinks that takes significantly less time now.
I guess this has become part 3 of my "minimize everything digital" series [1][2].
I self-host Nextcloud on a server at home, which backs up to a pi with an external HDD off-site. I manage these backups with Borg. For all of the things not in Nextcloud (config files, etc.), I also use Borg to backup to the same Pi. I just recently had to rebuild my entire stack, so I am confident that these backups work, although I have not sufficiently automated restoring from them.
I have used my system defaults with a few extras for a while, but am just now experimenting with something more custom. At the very least, I renamed all files to be lowercase to make command line navigation easier. I also removed some directories I don't use, and added a few top level that I felt were helpful (I will cover these in answers below).
I have
~/downloadset to be my default, but I have my browser ask me where to download every time. It's annoying 30% of the time, and useful 70% of the time.Both, really. It depends on the file and what I need to do with it.
Mostly camel case, but the files are really not as standardized as they should be.
I have
~/code/in my home folder for code repositories. Since they are almost all git repositories that are backed up elsewhere, they are not usually included in my backup regime.I don't have a good answer to this yet. One I have heard, but not tried, is setting your home directory to be read-only, but I don't know how much that may or may not break things.
I use Calibre for e-books, Airsonic (soon to be moved to Funkwhale) on my server for music, and Jellyfin on my server for videos. All images and documents are in Nextcloud, and I manage those mostly by hand.
Mostly I use a file hierarchy in Nextcloud, although not all of it is well organized. My school folder is well organized since the input data is nice and hierarchical and I use it regularly. The hierarchy goes:
School name >> Semester >> Class >> Project. My photos are partially organized. The old ones from before my organization are a mess, but any recent ones are synced into a folder based on the device they come from, and they retain timestamps. My documents are a mess, I have some folders to help, but nothing cohesive.I also have a
~/programs/directory in my home folder where I mostly keep these. Really, this folder is for code I do not intend to hack on, just to use, so there are some compiled git repositories in there as well.~/code/is exclusively for source code I want to look at or manipulate, and is mostly git repos and some extracted tar files.I have a folder called
!!! Dataat the top of my profile folder. I keep everything of value in there, in case I find myself having to switch platforms again. (If I can afford to, I would also transfer whatever savegame files I have lying around. If not, at least I have everything else stored.)!!! Datahosts a handful of folders – mostly categories by which I handle the files – as well as!!! Inbox, again at the very top.Everything that I don't use immediately, I put into the inbox folder. It can be cluttery at times because it's allowed to be: it's a kitchen sink of all the files that are between destinations. The inbox folder has a few subfolders of its own, mostly for reduction of visual clutter, but also for cleaner transition between itself and the destination folder. Images to go
!!! Inbox\@ images, videos go to!!! Inbox\@ videos, text files of varied kinds go to!!! Inbox\books and suchetc.I have a vague convention on naming folders: if you want something at the top (perhaps because it's a parent category of its own), you prepend exclamation marks and a single whitespace to its name. If it's a subcategory (like
@ imagesis for!!! Inbox), it gets at-signs and a single whitespace in front. There's also the#octothorpe categorization, which is roughly for folders that are important in themselves (like!!! Data\# Writing, which stores all of my tales and some of the notes). None of it has been formally defined: it's something I follow on an intuitive level, retooling the system as necessary.Which, frankly, describes much of my workflow. If something needs a separate category because it's gaining traction as a fraction of the contents, it gets a category. I have
!!! Data\Markdown Documentsbecause I have enough of those. I have!!! Data\Textbecause I make a lot of plain-text notes that fit nowhere else.If I need to write something down now, I use a plain-text file of vague, ever-evolving formatting, by calling a set shortcut to the Notepad app. I then save it to the desktop: it serves as an inbox for notes, of sorts. That way, I can later access the notes in an instance, without having to trawl through a hundred files.
Which is to say: the inbox is there because I never categorize immediately. It may take a month before I even touch the folder again. Until then, the files are there, and I'm safe in the knowledge that eventually, I'll get around to it, because I always do.
Frankly, the only reason I don't use Indigrid for the immediate notes is because it doesn't have that feature. I've spoken to Mark about it, and he's given me no signal of interest. (To be fair, he never does: he probably just jots it down and implements it down the road, like he did with my virtual-space suggestion.) It's something I'd like to do with Intergrid, now that I also have the idea of making it into a new-tab browser extension.
Between the plain-text notes and the Intergrid intervention, I found myself using a different browser extension – My Notes, which turns the new tab into a plain-text note-taker – rather comfortably.
As a sidenote, I also store the installers for apps I find myself using with some regularity in a separate category within the data folder. I had, at one point, found myself having to transition between fresh installments of an OS due to failure in HDDs and SSDs, so I figured: having the batch of apps ready without having to find them later on would prove beneficial.
As for backups... I am woefully behind times on this. I save it all maybe twice a year, which I recognize is egregiously-insufficient, especially considering the trouble I went through to consolidate all of my valuable digital resources. I'm looking for a solution that, once set up, will require minimal action on my side to perform. I have an external HDD I use for these backups, and I'd prefer to have at least a local-first solution, and ideally one that backs-up both to the HDD and to a reliable cloud server.
I’ve set up my life such that anything of consequence is within a cloud service - I use Dropbox heavily and Google photos for pictures and videos I make, Google drive for most documents, Evernote for notes, source code on github, etc. I don’t have significant media now that streaming is so common, so I got rid of a lot of my local storage needs.
This means that I have no need to “back up” anything, and it is already accessible via mobile as well. This also means that setting up a new phone is super easy, there’s essentially no data I need to worry about copying over. The only challenge here has been text messages, since I don’t trust iCloud as it seems crappy.
A word of caution: I recently reclaimed all my data from Google after relying on it for storage and backup of my files and photos for years. And it sucked. Google Takeout will create archives for download, but it takes forever and is not necessarily organized nicely once you decompress them. If you have a lot of photos, it is very cumbersome.
Also, as someone else mentioned, it’s good to have physical backups as well ;) I have unlimited storage on Google through my uni, but for lots of reasons I’ve been working toward self-hosting Nextcloud and other similar services instead. I highly recommend it!
Edit: typos!
Yeah, it’s a point well taken. For a while I was actually double syncing to both google photos and dropbox for redundancy. Maybe I should do that again.
I have been tinkering on my backup system for months now. Right now, I am using Backblaze B2 and rsync; its dirt cheap and works. But the data isn't encrypted at rest and it is fairly fragile. So, I am slowly switching to ZFS as the filesystem and trying out snapshots. I am also looking at rsync.net for the storage, as I do love me some open-source tools. I am also looking into git-annex for a more versatile software component.
Using zfs as your filesystem will also add another protection layer to help prevent bit-rot through the zfs checksums. This is obviously not a backup strategy, but once you setup a scheduled zfs scrub (I scheduling mine via cron once a month) you can help prevent unnoticed bit-rot from happening that you may not even know when it happened to restore from a backup.
Newer zfs version 0.8.x offer native encryption if this is something you want to explore. You can also use zfs replication of the encrypted filesystem to a remote system and leave it encrypted such that the remote system cannot read the data. Basically you can keep a remote "untrusted" system up to date with your backups and not leave the key on that system.
Backups: external USB drives stored at my house, at my work, at the bank.
Storage: One folder with all my stuff less sensitive stuff grouped into subfolders called Photos, Music, HTML etc... Sensitive data in an encrypted image.
I use the file browser to move stuff into my one folder/ encrypted image.
Notes are in a word document or in the native notes app.
I dont have sandboxed apps, but now I am thinking I should....
I built my own sever at home which serves as one location for backups and important documents. I still keep documents, like for school, on my computer of course since I can't always just be connected to my server. I also backup to the cloud with Backblaze. Lastly, I have a physical backup drive where I use Carbon Copy Cloner as I use a Mac.
On my computer (which is a laptop), I organize everything by folder and relevance. So there is one large folder for "school" which then has sub-folders for different grades which have sub folders for subjects, etc. I also use alfred on my mac which allows me to navigate and search for a folder or file very easily.
I am thinking of setting up syncthing on my mac with my server to make it a little easier. Or I could do it through nextcloud. Not sure yet, and if anyone has any experience with a setup like this, let me know!
If you have any more specific questions, let me know. Happy to answer them!
I've gone through numerous iterations of data re-organizations over the years, but lately I'm going through a significant digital minimalism phase. While i still want to preserve milestone and keepsake stuff, I'm getting to a point where I think more and more stuff is ephemeral and really need not be archived/saved/preserved.
That being said, i have always found it valuable when i learn of how others manage their data, so here goes my setup...
I'm not very great at backups. I use dropbox and onedrive, but back up the data from both of them to a server in my house using samba. This obviously should be upgraded to a more automated style of the 3-2-1 method of backing data up...so that my future project right after new years. I do like the fact that both dropbox and onedrive have a "revert" function where you can turn the clock back a day or some to "go back in time" in case you totally kill some essential files. Is it a great back up strategy? Of course not! but i do like it, and have used it once.
BTW, currently I use dropbox for my most essential - and core - files. This comes to about 1.75 GB, so its not a big deal. I use MS OneDrive for my family because it comes with the MS Office 365 subscription, and the storage is quite generous...So i store my family's photos on it. Functionally speaking, i have not seen a difference between onedrive and dropbox...And the ONLY reason i even use dropbox, is because there is no official onedrive client for linux...and linux is my daily driver OS. But if/whenever microsoft creates a native linux client, because of lower cost, i might just move everything to onedrive. I do self-host my own nextcloud server...but i've not run sufficient tests to ensure preventing data loss...also, my nextcloud server is hosted on a cheap vps, but storage there is not as cheap (at least for the amount of data that i'm saving.) Nextcloud would be my preferred storage for local and remote file storage and synching/sharing...but i'm not there just yet. for now, its a combo of dropbox and onedrive.
On linux i do place everything under my /home/$USER directory hierachy, but since i use dropbox, it is really all under /home/$USER/DropBox/...etc... At all my dayjobs, i unfortunately get issued a windows machine, so just have everything under whatever DropBox folder gets set up by the dropbox app - i think thats under the typical Windows OS user dir. hierarchy, but i don't exaclty recall, nor care since its all under dropbox.
I do use whatever default download folder my OS or browser set. The stuff there is always considered ephemeral, so i care nothing about it after i move the files away from it.
I navigate both from the file browser and command line - it really depends on what I'm doing. But as i noted above about going through digital minimalism, i find it refrreshing navigating through command line more and more. Though this isn't possible or convenient with all desktop apps. Again, it depends.
I name almost all of my files and folders lowercase and almost never with spaces. I prefer hyphens/dashes to separate the terms used in a filename/folder name - e.g. some-file-name.txt, some-folder-name, etc. I find this easier for scripting, etc. Also, i avoid using special characters within the filenames for fear the Window OS side might mess up a file or something. Also, whenever needing to use dates in a file/folder name, I always use the iso format of YYYY-MM-DD, and that solves my sorting needs quite easily. If - as others have noted - i wish to somewhat permantently sort some files (to have them hsow up at the top, etc.), then I'll prefix file names with numbers. One file that almost always exists in my folders is the 1-README.md (or sometimes 01-README.md) file...This is a convention i use to help explain "why" i have the current directory in existence. Part of it is to help with my archival reasoning, another part is WHEN i die, it will help my family know what stuff is contained in the current dir. With my digital minimalism i find les use for the 1-README file, since I've recently greatly reduced complexity of my data org....But I'm hoping this file will still help my future family.
I'll add that as far as my data org., here's what i've been using lately, though might reduce this even further in the coming days:
I hope my comments help. I'd be happy to answer any other questions you might have!
I do a local time machine backup and have Arq backing up my data to Backblaze b2. My documents are all in my ~/Documents folder synced to a VPS using Syncthing so I can access the files on the go as well.
Code and notes are stored in public and private GitHub repos, I do this for notes mostly so I can have them available on the go as well with WorkingCopy on iOS.
I edit the files on my mac with nvUltra (beta). I switched from NValt to nvUltra and NValt wasn't capable of having a folder hierarchy so I prefix my notes with somewhat of a tag or category. But I mostly use the search functionality.
I actually like my setup but somehow I'm tempted to switch to a hosted solution for my notes like google docs, notion, onenote or office 365. I don't want to take care of the management of things like Syncthing to keep things in sync but I also don't want to give up my files to a weird format like gdocs.
I use Google Drive for as much free cloud storage as I can get, and keep the other stuff backed up onto a separate drive. I don't use any automatic cloud backup programs, because I don't know of any good ones. Anybody got a cheap recommendido?
I, uh, really should have backups. But I don't really do anything as much as I should.
Right now, my current setup includes a basic SATA SDD which I use as a boot drive plus applications. I combine that with two HDDs; one is used specifically for video games (they just take up too much space!) and the other is used for my actual important data. Those are in a really simple hierarchy based on what they actually are.
The good thing about separating my data drive from the OS and applications is that it means that it gets significantly less access and wear, so it will last longer. And as it grows, I put it on larger and larger drives, so they are getting replaced over time. So even if I am not doing backups like I should be, I am still doing proactive data maintenance to prevent catastrophic failure.
I do use the default download location for most things, but that's because they're often ephemeral. I'll go through them every now and again and delete them.
I plan to build a NAS sometime in the future, but honestly I don't have the money to buy a good solution (like a Synology system) or time to implement a cheaper version.
The 3 backups rule is definitely the normal advice. One local copy, another local copy on a different machine or media, and another copy offsite. I'm doing OK by this standard, but I've got some work to do. I bought two 10TB EasyStore HDDs from a Best Buy sale about a year ago, shucked one and put it in my PC, and kept the other in its external enclosure connected to my PC. I have Veeam for Windows backing up the internal HDD to the external HDD every night automatically. Otherwise, I have files strewn a bit across OneDrive (have 1TB as part of O365), DropBox, iCloud, and other PCs. I really don't have enough cloud space to truly follow the rule of 3, since my local data is ~2TB, so I need to figure that out shortly. Maybe BackBlaze?
I backup important things on my second PC and external storage devices and add more redundancy with rare things that absolutely need to keep their backups.
I have very particular and practical naming conventions for music and media, but don't worry too much about anything else as it will be organized into an appropriately named folder and easy to find. Which I use a program called "Everything" that lets you type and search in real time, very handy tool for finding things on your PC.
For folders I avoid the OS provided folders as they generally have shitty preset parameters on them compared to a new and independently named folder. I also like to keep my storage drives separate from the OS drive, first so that they don't all crash at once and corrupt a bunch of files and second so they are easily swapped into another PC should I have other hardware problems. Also less shit on the SSD should prolong its lifespan a little more.
I use MediaMonkey for a music player, MPC-HC for a video player. ninite.com has a good selection of free programs that you can choose from.
I don't backup anything now. Only the most important files (encrypted files with passwords).
I use custom directories.
~/data is my HDD with the bulk of what i have
~/data/personal with personal files like paychecks, journal, etc
~/data/media/{music,videos,pictures,books,other}
~/download/{torrent,soulseek}
~/bin with my custom programs and scripts
~/dev for programming projects
Everything is lowercase. My music is organized in ~/data/media/music/artist/year - album folders.
a few removable drives, no fancy formatting needed. I know where they are