How would you theoretically go about mitigating the potential near-complete loss of archived audio and video media from 1990 to 2020?
This article from last year provides an alarming look into the woes that media preservation (specifically audio and video) is facing this century due to a content explosion that shows no signs of slowing down. It’s not a new problem, as journalist Bill Holland showed nearly 20 years ago (warning, it’s a long read).
To summarize: In the past, many predecessors to existing media studios did a bad job of archiving their collections of recorded material. In some cases they actively destroyed or threw out parts of their catalogs to make way for new material. This wiped out portions of the available media to be preserved, especially the older stuff. Now that most studios have improved their archival practices though, their remaining catalogs are facing a new foe: Moore’s Law.
The problem with LTO (tapes) is obsolescence. Since the beginning, the technology has been on a Moore’s Law–like march that has resulted in a doubling in tape storage densities every 18 to 24 months. As each new generation of LTO comes to market, an older generation of LTO becomes obsolete… Already there have been seven generations of LTO in the 18 years of the product’s existence… Given the short period of backward compatibility — just two generations — an LTO-5 cartridge, which can still be read on an LTO-7 drive, won’t be readable on an LTO-8 drive. So even if that tape is still free from defects in 30 or 50 years, all those gigabytes or terabytes of data will be worthless if you don’t also have a drive upon which to play it.
If the worst case scenario were to happen, this is apparently what it would look like according to “a top technician at Technicolor”:
“There’s going to be a large dead period,” he told me, “from the late ’90s through 2020, where most media will be lost.”
But not everyone is that worried, the article also includes this counterpoint,
“Most of the archivists I spoke with remain — officially at least — optimistic that a good, sound, post-LTO solution will eventually emerge.”
/u/boredop and I have been discussing the implications of this in the thread they posted a few days ago about a John Coltrane release, and in the course of that discussion they provided that second link to Bill Holland’s multi-part investigation (thanks!).
So my question is this: What direct or indirect measures would you theoretically take to prevent or mitigate the loss of the vast majority of recorded media from 1990 to 2020? Should any measures be taken to preserve these cultural artifacts?
By direct measures I mean innovations to physical archiving or storage methods. By indirect measures I mean public awareness, strategies for choosing what to save, workarounds, etc.
Excellent question. It's ironic that even while many consider libraries obselete, they have never been more important.
Their original function, to archive and loan physical data/information to the public, is not different from their inevitable future function, to archive and loan digital data to the public.
There's also a lot of books that aren't available in digital format yet so they're still essential for that purpose.
It hurts my heart to think of all the libraries that have been purposely destroyed or otherwise lost to history :(
Well, history is ongoing. Mosul, Timbuktu, various important libraries had been destroyed not so long ago.
I know, but it’s terrible to think about the amount of human progress and knowledge that was and will be either deferred or completely lost to time in either direction.
The Library of Babel by Jose Luis Borges changed my perspective on libraries and what it is they're meant to accomplish. It made me consider why humans excerpt information in general.
So did someone take a short story and make a website based upon it?
It'd be somewhat neat to fill those shelves with something like libgen, although entirely useless lol.
This is a large part of why I consider some torrenting communities to be so valuable.
There's a site with approximately 680,000 music releases (an individual album, EP, etc., not counting various reissues, source formats, etc.), with the majority actively seeded and available in lossless format, all meticulously organized. It's been running for barely a couple of years, with its predecessor having an even larger catalog until it was shut down due to legal issues. Same thing with a movie site that currently has 8,400 titles, and one for TV with thousands, if not tens of thousands.
I'd be surprised if every library in the US combined had anywhere remotely near the amount of content that these sites have. They have some flaws of course, but I can't imagine anything better that currently exists.
So one suggestion would be, let these places flourish freely.
You bring up an interesting point that's paralleled in Bill Holland's multi-part series, which is that bootleggers have actually played an important role in the pursuit of media preservation at times when studios didn't know the value of their own collections. In some cases, if it weren't for bootleggers, there are important recordings by very famous musicians that would have been lost forever.
Anyway, as for digital preservation, /u/rib pointed out that the Internet Archive preserves a lot more than just web pages. But copyright and intellectual property rights seem to be something they're very strict about, which is where sites like the one you mentioned come into play. They may be seen as unsavory nowadays, but in the future they may end up being lifesavers.
I think this is the right idea, at least as a stepping stone toward the true solution. The internet appears to be evolving into a more robust version of its original form. This emerging network will be hardened against censorship due to a number of factors:
I think the puzzle pieces are coming together for a new system that will solve the archival problem and be reasonably future-proof. Of course, there are some obstacles in the way:
I believe the first will solve itself, in time. Intellectual property is an archaic, obsolete form of protectionism and it won't be able to survive the digital age. It will take time though; stakeholders aren't going down without a fight. Also, we should realize that a lot of things that exist today could never have been made without copyright protections — many careers, large film/music projects, a whole celebrity culture. The recording industries will adapt and slim down accordingly, but not without some pain along the way. People will look back and view this decade as a "golden age" before everything was turned on its head. I think it's unfortunate but unavoidable.
The second is only going to be fixed by demanding an open competitive market for internet access. Net neutrality legislation is a bandage that doesn't address the root cause. There is no good reason why people who live in a given metro shouldn't have 10 or more options for getting online. The technology exists for numerous alternatives but the municipal agreements entrenching the established telcos stand in the way. Those agreements need to be declared unconstitutional and thrown in a fire ASAP.
I'm not sure about the third item. The big companies have a huge advantage that can't be overstated. I'm optimistic that falling tech costs will empower people to roll their own, but I'm not convinced the incentives are in place to make it worth anyone's while. I'm an AWS customer and their service is far better than anything I could manage myself or trust to a smaller company. I hope one day that will change, but to me this is the biggest wild card right now.
Great thoughts. One question and a bit of disagreement.
Do you think cloud storage is essential here? Take the music site I mentioned. 691,386 releases are maintained by 30,313 users. If issues 1 and 2 are solved, presumably the populations of such places would balloon. I don't have much expertise in storage solutions, and I'm not even sure if this wouldn't be defined as some form of cloud computing, but do you foresee issues in simply scaling up the system as it works currently (local hosting distributed via torrents)?
I'm wary of involving any profit seeking entity in such matters. When you say people might be able to run it independently but you're not convinced it'd be worth anyone's while, in what sense? Financially?
Unfortunately, the site doesn't show just how much data there is, or how much is transferred on a monthly basis, but doing some rough estimate math, I'd say all of those releases in FLAC would be about 500 TB. I wonder what the costs would be, and whether it's something that can realistically be crowdfunded.
I'm conflicted about net neutrality. It's a bandaid as you say, and I think it only further entrenches the broken relationship between public and private. What you say is exactly what I wish would happen, but I don't see it being fixed anytime soon, especially with there being so much overlap between the ISPs and legal ownership of much of the media in question.
I don't know if my view on IP is more extreme than yours, but I think it should absolutely abolished without any sort of 'lighter' form of legislation to replace it. Just outright end the entire idea of there being any sort of forced control over infinitely replicable digital data. I also think this is inevitable, at least on a practical level. I'm not as optimistic though on the government getting out of the way anytime soon, if ever, and sadly, there's arguably decent justification for all of it. I think this will be the biggest hurdle by far. At least until now, it's the only thing that's stopped these places.
In any case, it's just so disappointing to me. For such an absurdly low price (given what's being gained), in such a short amount of time, we can give every person with an internet connection near instant access to nearly every piece of music, film, broadcast (and eventually text) that humanity has ever created (besides the stuff already lost), on platforms more elegantly organized than any other I've seen. A practically self sustaining library for the sum total of human creativity. And we get Hulu...
Putting aside the illegal nature of these sites, their primary purpose does seem to be the preservation and sharing of as much media as they can find which I actually encourage. They are clearly not doing it with ill intent and simply want to catalog as much as they can. I also believe they should be left to flourish freely, especially considering that the data can be shared in such a way that pretty much every piece of music is stored on at least one users computer somewhere. Centralised data banks and libraries are confined to a single building or server banks which can easily be targeted, whereas sharing the files around the globe effectively ensures the safety of said files.
Not sure about that old media but there's archive projects for digital/digitized media like https://archive.org/. You can upload stuff to the site provided it's not going to be DMCA'd down.
I love the Wayback Machine, I use it almost daily for internet sleuthing, and some of the most important (to me) info I've found on the net has been through the wonders of that internet archive. I didn't realize how heavily they archive multimedia though.
Having said that, only some of the studio media catalogs in question have been digitized, so that presents part of the problem.
What's the your favourite thing you've found on the internet archive?
Hmm it would have to be this 12-part introduction to quantum theory by Andrew Thomas. I found it when I was looking for info on the block universe theory of time.
In a way I think it's kind of cool that there will always be a collection of music, or any media that has been around for the last century, that will never be recovered and is simply lost to the ages. Just like Indiana Jones searching for the Holy Grail, I can imagine in a few hundred years we'll have films about people searching for Wu-Tang Clan's Once Upon a Time in Shaolin CD.
I'm imagining a future where hundreds of years from now the works of musicians like Nirvana and Dr. Dre become tall tales, mythology that still affects culture, but with no clear origin and no concrete proof of ever having existed because it all want up in digital smoke lol.
Humor aside though, the thought of so much unreleased and lost music is not a happy one for me. It means that so many beautiful multidimensional ideas in musical form will never see the light of day again, confined to one tiny period in time and space instead of having been able to reverberate across those dimensions.
A little off-topic but that reminds me of a faction in the game Fallout: New Vegas. In the post-apocalyptic remains of an old school of impersonation, some wastelanders have taken to calling themselves the Kings. They've built an entire ethos around the personality of Elvis Presley: they dress like him, talk like him, perform like him. But they don't really know anything about who their hero actually was, and assume "the king" was an actual title rather than a showbiz nickname.
LOL he became Jesus 2.0 – the man, the myth, the legend. That's actually a perfect example of what I meant, though, so not off-topic at all!
I definitely do see the sadness in there being music that will never be found again. Question is surely those pieces of music that have survived the ages are those that actually deserve to survive? If a song is lost I can imagine it was most likely because it didn’t appear special or important at the time. This is not me saying that some music is above others, but when some music is held in higher regard above others, then it’s chances of survival are increased.
This of course mainly applies to older music, as most music created now days is backed in several locations, spread about the internet and would be easy to locate.
Unfortunately societies are not always adept at recognizing important music as its happening. For example, the early blues musicians in the US. Due partly to their race and partly because of the prevailing tastes of the time, these musicians were under-recorded and the records they did make usually sold poorly. (Also, one of the main labels for these records, Paramount, went out of business in 1935 and destroyed all of their metal masters.) That leads to situations like Robert Johnson, who only had two recording sessions which were not widely recognized until decades after his death. Or even more tragically, Johnson's contemporary Willie Brown, who only got to record six songs. Only one of his records (with two songs) is known to still exist today, and only three copies of it have survived! The other two records are gone, possibly forever. Or Geeshie Wiley, whose recorded a classic called Last Kind Word Blues. It's believed there are less than 10 surviving copies! There are plenty of other brilliant records that barely survived with fewer than a dozen copies extant. It is due to a stroke of luck that these records exist for us to listen to today. There are others that have never been heard by modern ears. Perhaps they'll never be found, but the search continues.
Society does not have a great track record of taking care of valuable pieces of art and other creations in general. I can imagine that there have been so many wonderful masterpieces simply lost to the ages due to accidental discarding, poor treatment or as you mentioned because of society's feelings towards the artist and art at the time.
I’m glad you brought this up because one way or another, humanity’s going to have to figure out some type of strategy for the collective media catalog it’s amassing. You’re right that the default sort parameters are based on popularity and thus commercial viability, but once that music stops being commercially viable, its past popularity becomes less important and sustainable to the people in charge of financing its continued archival.
So you end up with a situation where people pay less attention to a lot of the older popular media as more new popular media is created, and there becomes more of it to store. So even within the context of only preserving the most popular media from each time period you choose, you end up in a situation where you’re scrambling to keep up with archiving an exponentially growing collection of material.
It’s harder to gauge the impact and popularity of media when the amount of it out there continues growing exponentially. Is it fair to say that present popularity is the best way to determine what gets saved if popularity itself becomes diluted by the sheer amount of existing media over time? And beyond that, aren’t there media that deserve to persist despite their relative lack of popularity?
Even though yes media production is expanding at an exponential rate, so are storage solutions and compression algorithms that allow this increasing influx of data to be stored as it grows and grows.
I guess when considering what media to save and what to discard both the popularity and the impact of said media has to be taken into consideration. In most cases both properties are linked, an example would be the Star Wars Original Trilogy. The films were not only extremely popular but had a huge impact on the genre. There may be other cases where media initially did not have a large popularity rise but still had impact on the world, one that may not be as obvious but still left an impact nonetheless.
Whether a piece of media is worth keeping or not is still essentially a subjective choice; some people may love one song, others will hate it. Trying to find an objective property for a piece of media on which to decide whether it is kept or discarded is near impossible as its entire value is based on the emotional response it creates, not the physical media itself.
Right, but even then you have to keep up with transferring that archived data from one storage solution to another, as the first article explains:
I wouldn't go as far as saying it's that close to impossible. For example, another way you could determine cultural impact aside from popularity would be to map how many derivative works or other cultural artifacts branched off from that initial kernel or seed of media.
I agree, the transferring of data will most likely be the bottleneck of the process, not the storage itself.
That's a really good metric to measure actually, I think it would definitely ensure great works of media are kept because for the most part, these great works changed the game in a significant way. I like it, it's a property that's a good mix of both the objective and the subjective.
I’m glad you think so, I’m preliminarily working on a project idea that does that with music!
Wait so you’re creating some kind of algorithm?
Eventually yes, that's the plan.
I'd very much be interested to see how the algorithm develops. I can imagine the most difficulty will be in whether you decide what other pieces of media were inspired from another, especially if the creator of said inspired media never explicitly said they took inspiration.
At its most basic, it started as an idea is to map 'root' songs and the covers and renditions that they spawned, essentially as a way to try to predict what criteria or threshold of attention needs to be met before a song becomes a cultural standard.
Interesting, I'd definitely like to see how it evolves and it's progress!