28
votes
Are there any datahoarders in here?
Datahoarders are people who will keep an absurdly large amount of data on a number of large capacity hard drives. That data can be anything from 4K movies, family photos and recordings, archives, YouTube channels... anything really. I find this practice to be intriguing. Do you feel like this description may apply to yourself?
If so, do tell us more about your endeavor: do you collect anything you can get your hands on or do you have a more specific aim? Do you share any of it? Do you have a particular setup? That could be hardware, software or some cloud subscription.
My file server currently contains...
That's just what's been sorted into a folder structure that's easy to browse and import into various programs. There's a lot more I haven't fully imported into the collections yet, mostly due to lack of an automated way to get it done - especially music. It's all sitting comfy on a 32TB RAID-6 array of WD 4TB drives inside a synology NAS.
I own a big chunk of the music/tv/movies on DVD/BR, and there's another 1200ish movies I haven't ripped off the physical media yet.
If I'm browsing the web and I see something I like, I download it and watch it after it's in the collection. If I really like it, I'll buy it to compensate the creator - unless I've got an issue with that company, then it's second hand purchases to deny them revenue. When I'm running a bit low on space, I purge whatever it is that no longer holds my interest. Last purge was every single marvel/dc television show and the vast majority of their films.
This is what I do with my music collection. Granted it's not quite as impressive at around 100k songs and only 1TB (I convert everything to mp3@320kbps). Got quite a bump from my recent interest with classical music. Some composers were really prolific. I buy new stuff from many of my favourite artists, but there's so much music I want to hear, I'd quickly go broke if I bought everything. Still have some mixed feelings about it, though.
I don't care for streaming, I much prefer having the files on hand so I can stick them on whatever device I have without worrying about them disappearing. Used to have automatic cloud backup, but a 4TB external is enough to hold everything important, so I thought it was bit of a waste.
There are many ways to automate music collections. If it's just a lack of tagging, musicbrainz Picard is what you are looking for.
LFLT: https://picard.musicbrainz.org/
TuneUp is really good too, especially if you use iTunes or WMP... though you have to pay for it, and it isn't FOSS. Back when I still bothered to maintain my own mp3 collection it's what I used, and it only struggled to identify the absolute most obscure stuff I threw at it.
Thanks for linking TuneUp, that's one I haven't tried yet. I don't mind purchasing a commercial product for this if it actually does the job. I'll take the trial for a spin and see if it's got the chops to make a dent.
NP. I haven't used it in a few years now, so it's definitely possible it's gone downhill since then, but it was pretty solid last time I used it.
p.s. If you try the demo and it works out for you, rather than you buying a full license I can just send you mine instead, so that you can transfer it to your computer. I very much doubt I will ever use it again so it's kind of a waste me holding on to it. So let me know if you're interested.
I tried that one last year. Picard is only functional if it knows the artist/track's sonic signature. That means it only works on artists that are popular enough to have already made it into the database. Collecting popular music is definitely not my hobby, that's the music I'm least likely to have or even know the artist's name for the asking.
Bulk tagging is pretty easy, several apps can do that job. Mediamonkey and Tagscanner are my favorites. My problem is I have mountains of music (dating back before alt.binaries.mp3 was a thing) that I can't identify and that no program can identify either. I could do them by hand, but that would take literally years.
I am hoping that in a decade or so something like Picard is actually up to that task. I've been waiting a very, very long time for a tool that can finish this job.
I also have a good portion of my music collecition that many would categorize as definitely not popular music. As a side hobby, I'm teaching myself python...and one project i hope to undertake when my skills are far better is to use my music collection as training data for some set of machine learning algorithms. Sort of have it take a swipe at my music collection, and try to automatically tag my music...and then i would check a few results to review accuracy, have it go several more rounds, and iterate, etc. I know there are perhaps better tools for tagging music...but much like you, my music is not as popular enough to take advantage of existing databases...plus this project is more for me to augment my python skills primarily and learn more about machine learning secondly...with the tidying up of my music collection's metadata a third benefit.
I don't hoard a tremendous amount of data, but I do hoard one particularly niche thing. I created /r/coversongs and ran it for almost a decade, so I have ~24GB worth of my favorite YouTube coversongs downloaded, archived, and backed up, since coversongs are notorious for suddenly vanishing on YouTube (both due to copyright strikes and the artists deciding to make them private for whatever reason), and I really don't want to lose access to any of these songs, like I have to so many others in the past.
About a year ago @suspended also did me a huge favor and remastered ~40 of my favorite YouTube coversongs with the poorest sound quality, making them much more listenable than they were originally, so I don't want to lose those copies under any circumstances.
As for my hardware, it's nothing special TBH. I just have the data mirrored on 2 drives (1xSSD and 1xPlatter), and automatically uploaded to the cloud as well.
p.s. Since YouTube is where I tend to listen to music, whenever one of the coversongs I have backed up suddenly disappears from there I reupload it to my own account as unlisted so I can keep them in my playlists. ;)
So you download the audio off from YouTube? Is the audio quality good enough in that case?
youtube-dl can do wonders. It can download multiple streams and mux together the highest quality of both.
I download the audio and video in most cases, since a lot of the coversong videos are just as entertaining to me as the music itself. As for the audio quality, that varies a ton, especially since a lot of the coversongs are ancient, and were made by complete amateurs (at the time) who only had access to shitty home recording equipment (e.g. just a laptop mic and webcam in a lot of cases)... but I am more interested in the soul of the music, and witnessing the potential of the artist, so generally don't care about the bitrate.
E.g. To give you an idea of what I mean by that, IIRC this is one of the first videos I submitted to /r/coversongs way back when, and still one of my all-time favorites, despite the low quality audio/video and background hiss: https://www.youtube.com/watch?v=HMrqBldlqzA
p.s. Since then, First Aid Kit have made it reasonably big too. :)
I know I’m veering off topic, but wow, they just absolutely nail those harmonies and it’s beautiful! Honestly I think the stream in the background just adds to this, what a fantastic cover.
Yeah, their harmonies are always lovely and spot on, IMO. Another of my favorite videos featuring them harmonizing, is when they performed their song Emmylou live in front of Emmylou Harris at her Polar Music Prize award ceremony, which brought tears to her eyes. :)
https://www.youtube.com/watch?v=Hi5A9OCAyIk
Wow yeah! I am normally not a fan of music like this, but that cover was beautiful!
And that's such a cool thing to hoard, I've also seen quite a few great cover songs disappear from the internet. Actually I posted a cover on youtube yesterday and it got instantly copyright claimed because it apparently matches some random some song I couldn't even find. If it was at least the song I covered, but no. So I am not even surprised songs just disappear from youtube!
I'll start. I don't think I'm a hardcore hoarder but I do have a tendency to cling to things, be it physical or digital. When it comes to the digital side, it's great to just store pretty much endlessly, as the limit keeps being pushed by ever expending total capacity on the hard drive market.
I've mainly hoarded music, particularly game soundtracks that may be hard to find. This started back in highschool. I do use services like Spotify but I don't feel comfortable knowing the music I love could disappear after a licensing agreement gone awry (and also most game OST aren't available on there). Of course that does mean gigabytes of FLAC files I don't listen to mostly gone unused. I also try to backup my personal files, documents and coding projects. Both music and files tend to be precisely sorted into directories.
I'm a bit of a hoarder, but it does follow some logic:
Transient media, like 1 or 2 tv shows I can't get without paying another 10+ a month for get dumped on a disposable platter.
I do, however, preserve anything that would be classed as a 'desert island' entertainment. Favorite movies, music, tv, books, and games get archived to a mirrored drive and will soon be backed up offsite as well.
One reason I do this is because I don't like being beholden to always-on internet. You never know when stuff might disappear, or you can't have internet access for one reason or another. Or if your budget crunches and you can't rely on paying for streaming services.
There's the usual backups of personal stuff, but even then I let most of that be transient until I had a kid and decided I actually do care about backing up their photos. I take about 1000x more photos and videos of my kid than anything before in my life.
Yes! about 55TB raw in total. I'm a homelabber as well so I hoard all kinds of hardware too.
My setup is 3 ESXi hosts (a storage box and 2 intel NUC10i7) and a different mix of VMs + k8s cluster for hosting applications -- mainly Plex and satelite services like sonarr, radarr, nzbget, nzbhydra, torrent client, etc.
I own a ton of 4K movies, games, tv shows, documentaries, anime; anything you can think of. Youtube pissed me off with their incessant premium and youtube music ads on mobile so I ended up using Plex for youtube videos as well -- so I have a script that syncs the videos that my subscriptions post. I also use the space for personal backups.
I have a symmetrical gigabit fiber connection.
Those are Usenet software right? That's a realm I've always wanted to dip into, but it's was always pretty hard for me to justify paying for that when I didn't have a lot of trouble finding my way with other means. Of course nowadays I'm not so sure. Is there that much content on there? Would you recommend using it?
Yes, nzbget is like your torrent client and hydra is akin to jackett.
If you have access to a good number of private trackers, then no, I wouldn't recommend it.
If you have a restrictive ISP or one that blocks torrenting and you'd spend money on a VPN anyway -- yes, it's definitely an option as its an HTTPS download.
I was in the same place you are, just got usenet to see what all the fuss is about, didn't really need it as I don't fit in any of the two categories above. I paid for 2y in advance IIRC, so now I'm "stuck" with it. I use it alongside the rest of my sources and it's pretty good -- but, in hindsight, I shouldn't have spent money on it.
Oh, really? Why is that? Is it because the private trackers are better or for some other reason? I'm not in the two categories you've described so I was guessing Usenet would be better than nothing.
Don't get me wrong, you can find pretty much anything you want there. My argument is that you can find that for free anyway, whether on private or public torrent trackers.
Private trackers are generally better at niche content, but you'd be surprised what you can find on usenet.
Give it a shot if you like. Long story short you need an aggregator like nzbgeek -- there's free and paid options available -- and a usenet provider -- which is always a paid only option. I strongly recommend you set up nzbget + hydra + sonarr/radarr and not do it manually since it's a pain in the ass. Take a look at the r/usenet Wiki to find out more.
Edit: To expand on that, IMO usenet is the last line of defence for us pirates 🏴☠️. If your ISP starts blocking torrents or god forbid public trackers become inaccessible in some way, you've still got this niche thing that hasn't gotten any attention in 30 years.
I also see it as an alternative for when I'm finally fed up with the 14 year olds that moderate most private trackers.
That's the hardest part for me haha! I'm pretty sure I have a lot of duplicates but I'm never sure so I end up never deleting anything. I've found this tool so I'm going to give it a try. Do you have tips or other recommendations for organization?
I do ZFS datasets and it works for me, but YMMV.
I want to convert my server over to something ZFS-based. Bit rot has hit me before and it's not fun finding out the data you've stored is slowly decaying into noise. ZFS is built to soften the sting of that problem. You're never really immune to it, but ZFS can detect and correct it on the fly better than any other file system I'm aware of.
When using ZFS be sure to read this first. It'll save you some time (possibly lots) tuning the performance.
Is anyone here using ZFS on linux? How far along is the port from the original Solaris version, has it become a first class citizen of the kernel yet? I haven't looked in on this in a couple of years.
I'm running my NAS on BTRFS. I'm worried about the potential complications of ZFS not being included as part of the Linux kernel, something might break and corrupt my data. I'm much more comfortable using a filesystem built in. It does pretty much everything that ZFS does, as far as I can tell, and also has the advantage of allowing you to resize your RAIDs in a much more flexible way.
Right now I'm running BTRFS raid 1 on two 4TB HDDs. I'm thinking at some point I'll upgrade that to 4 16TB HDDs or something, but that's a lot of money.
I am, although I'm moving back to FreeNAS just because I want a pretty UI around managing it.
I would say we're close enough. Really the only complaint I have is that it turns on ZoL only features by default when you create a pool, and then that pool can't be imported into *BSD -- but you can disable those when you create the pool.
Good to know. As long as the data is stable/safe and I can get the root filesystem on ZFS I don't mind a little tinkering.
I've been using ZFS for years, originally on FreeBSD and now on Linux. I'm very happy with its stability on Linux.
It's not quite a first-class citizen on Linux due to license incompatibilities, but that's largely rendered moot due to so many distros having first-class support for DKMS. When you install a new kernel, it'll automatically download the matching ZFS sources and compile the ZFS kernel module locally. It works around the licensing issues while still making the process basically seamless for the end user.
ZFS-on-root is supported, and with the most recent ZFS update there's now native encryption support, so you no longer have to do LUKS layering if you want encrypted root as well.
I personally am still using XFS for root filesystems and ZFS only for data / storage drives, mainly because I like being on the beaten path for things like the Docker storage driver (which uses ZFS snapshots if
/var/lib/dockeris ZFS, and I have doubts that it's as battle-tested as the standard overlay2 driver).If you buy from ebay, there's probably a limited amount of copies. Torrents don't do that.
My personal rule is to only use this for large company software/files; small creators would definitely miss the money so l buy those.
Those resold copies still have to be bought somewhere. Those still give the creator money once per copy. A torrent practically pulls copies out of thin air, save for the original one that was used to make the torrent.
Way back, I ran a BBS that specialized in obscure texts. This was way before HTML was a thing and FidoNet was king. Then, it became increasingly easier to publish text and text content as an object kind of stopped being a thing (and rather impossible to hoard). It still left me with a weird tendency to hoard e-books across every possible topic that plays up every few months.
No. The only data I hoard is my personal photos, documents and videos, which cannot be recreated or reacquired if lost. I’m looking into finding some sort of cloud backup service as I’m a little nervous about something happening to the data which is all stored in multiple copies in the same physical location, my home. As well as the hassle of making sure I actually have everything backed up. I already had a big scare where I thought I had lost access to a hard drive which it turned out contained the only copy of something I thought I had backups of but didn’t. Thankfully I was able to salvage it and duplicate it.
Those concerns simply don’t exist with commercial media. Games, music, films, ebooks... If I lose them, I can get access to them again easily, and usually without any added cost. I find it enough of a hassle to deal with the data that I cannot replace if lost to want to go through it with data I know I can simply redownload. Even if it turns out in the future I will have to pay again for something I paid for many years ago, that is worth it just for the extra peace of mind. Maybe some ultra-efficient data hoarders with a practiced routine of disk-swapping and automated backup solutions don’t find that stressful and a waste of time, but I do.
By the way, check out backblaze for a set and forget backup solution. It isn’t the cheapest if you only have a small amount of data, but it is a good service.
I have also used Backblaze and had good luck with it. I once emailed Gleb (the CEO) to complain about something and he personally responded, so I feel like they're pretty responsive to their user's problems.
I've never had a reason to use their service, but seeing Gleb and other employees respond in depth to questions and comments on Hacker News posts about Backblaze has put them at the front of my list should I ever need the type of product they offer.
That looks like just what I need, thanks.
That looks like it might be just what I need. Am I right in thinking their proprietary software is the only way to interact with the unlimited service?
Not a deal breaker, but if so it'd be good to know how well it handles and keeps track of multiple volumes (internal volumes from three main devices, a separate network drive that they can all see, and 3-5 USB drives that get swapped around as needed). Don't suppose this is something you or @joplin have dealt with in the past?
They actually have 2 products. The one I mentioned is their normal consumer tier backup solution. Their other one is called b2. With b2, you can use rsync and other software, and you can backup anything arbitrarily. You pay based on the amount you store.
With the consumer backup, it just backs up everything connected to a single computer. It is unlimited storage, only limited by the amount of storage you can connect to a single computer. It will do removable devices just fine, but you have to plug them in again every 30 days. It will not do nas. That is by design, because they want you to use b2 on a nas. I think it will backup an iscsi lun from a nas, but that is a lot more work to setup. One of the best features imo is the restore options. You can download your data like you would expect, or you can purchase a flash drive or hard drive with your data on it, and they mail it to you. You can either keep the flash or hard drive, or send it back for a full refund of that cost.
I use their B2 product for backing up my music collection and photos. Their command line application is great for syncing folders. I also have a symmetrical gigabit internet connection and b2 is actually able to handle my uploads at about 85 MB/s, which means I was able to upload may 300 GB of photos in about 5 minutes.
B2 is great, if you’re willing to put in the work of building your won system around it.
OK cool, that makes sense. The price per GB on B2 is good, no question, but it does change it from a $60/year unlimited proposition to $300-400/year.
It's totally understandable that they need to make a distinction between personal and business use, and NAS is as reasonable a line to draw as any, but the irony here is that 90%+ of my total storage is internal or USB drives, with just a little under 1TB on the network to avoid duplication of things I'm working on between devices. While it is by definition NAS, it's just a basic consumer drive that supports ethernet as well as USB, so I don't suppose it would be badly against the spirit of things to use the USB connection on one machine and allow it to back up under the consumer plan. I'll consider further and likely give the free trial a shot - thank you!
Weldawadyathink covered it pretty well. I've not used it with a NAS, but I do have 2 external drives I use it with and it works great for those. They are always connected and always on on my desktop so I've never hit the 30 day limit. I have done the restore via hard drive mailed to me. Worked well. Even the hard drive they send you is encrypted, which is nice.
Good to know, thanks! It seems like it'll be worth a shot, especially given how satisfied everyone here seems with the service.
Yes! Please back your personal stuff up, and do it sooner rather than later!
When my first hard drive crashed, I had almost nothing backed up, so I lost a ton of data from early in my life. I don't really blame myself as computers were new and expensive and backups and drive failure weren't ever things I heard anyone talk about. I hadn't been exposed to the idea of backing up my files yet -- it simply wasn't in the zeitgeist. The hard drive failure took me completely by surprise, and it was devastating when it happened. One day it was working, and the next day it simply wasn't. And no matter how many times I rebooted it and begged and pleaded with it, it never worked again.
Like you said, personal stuff is irreplaceable -- even the tiny, incidental, inconsequential stuff! I would love to be able to go back and see those papers I typed for my high school classes or what random stuff from the fledgling internet I thought was worth saving, but it's almost all gone now (IIRC I was able to recover a few files from some stuff I'd saved onto floppy disks, as I've held onto a very few files from that time, including some middle school poetry -- yikes!).
On top of the file loss is losing the memory of what was even there in the first place. I've long forgotten what was on that drive, so both my files and my memory of those files are forever gone -- a double loss, of sorts.
In hindsight, I consider this a good thing, as it clued me in to the need for backups "early". Ever since then I've diligently kept my stuff backed up, and when cloud backup services came on the scene, I eagerly used them for the peace of mind they offer. I have since done two full restores from them for subsequent hard drive crashes. They have saved my bacon twice now! And each time it has happened, I've accumulated more personal data than the last, so the stakes were higher each time (and will only continue to grow as I acquire more and more of a digital footprint over time).
It's gotten to the point that I just expect my hard drives to inevitably fail, and many hard drives give no warning before they turn that corner. They just go. Cloud backups mean that I don't have to worry about that happening when it does (and it will), and they also give me peace of mind for other situations as well: what would happen if my house caught fire or someone broke in and stole all my tech stuff? My personal data is invaluable to me, so it well offsets the cost of the backup service.
I have a 4tb NAS with plans to expand to 12tb RAID 5 as I need space. I tend to hoard a bit of everything. Some media, but also stuff that might just be hard to find later like the Mario 64 decompilation project.
In terms of backups, at the moment I have a 64gb flash drive that I back up photos/documents and the data from the few web services which I run out of my house on to and then leave in my desk at work. I intend to sit down and work out how to use Backblaze for automatic option when I get a chance though because with the whole Coronavirus thing I'm about a month behind on my backup schedule which is freaking me out a little.
I do a bit of data hoarding here also. The fist project that started me down this path was to convert my large physical dvd and bluray media to a digital format. I spent a lot of time ripping my own personal media to build a collection that is now managed via Plex. When I began this project about 6 years ago, the bulk of my movies were ripped and saved, then I used Handbrake to convert the original source into a smaller h.264 encoded mkv with the original soundtracks (e.g. DTS & DTS-HD) and an extra AAC dolby prologic II track. This means I'm storing two copies of each with the raw and processed. I'm now considering removing the processed version and simply convert the raw ripped version into a bluray remux.
I've recently began ripping my 4K UHD media into 4K remux in mkv containers and making those available on my Plex. Those movies are now consuming a large amount of space. I store them in the same folder as the processed 1080P version to allow the option to pick either.
Plex is run as an Ubuntu server VM with 16 cores and mounts the network storage via SMB as a read-only mount to my NAS. I also converted all my music cd's into FLAC and put those on Plex as well.
My home lab hardware consists of two NAS servers with a bulk of HDDs.
NAS #1 is 12 x 4TB HDDs (1 x 12 disk raid z2 vdev)
This is usually powered off and occasionally powered on to rsync backups from my other NAS.
NAS #2 is 20 x 6TB HDDs ( 2 x 10 disk raid z2 vdevs)
ESXi Server #1 is Intel dual Xeon E5-2670 and 128 GB RAM (Intel Chassis and Intel S2600CP MB that was pieced together)
ESXi Server #2 is Intel dual Xeon E5-2670 and 128 GB RAM (currently powered off as it's not needed right now)