• Activity
  • Votes
  • Comments
  • New
  • All activity
  • Showing only topics with the tag "archives". Back to normal view
    1. Request for help: Backing up NASA public databases

      TL;DR: NASA's public Planetary Data System is at risk of being shut down. Anyone have any ideas for backing it up? Hi everyone, Bit of a long-shot here, but I wanted to try on high-quality tildes...

      TL;DR: NASA's public Planetary Data System is at risk of being shut down. Anyone have any ideas for backing it up?

      Hi everyone,

      Bit of a long-shot here, but I wanted to try on high-quality tildes before jumping back into the cesspool of reddit. I'm posting it in ~science rather than ~space as I figure interest in backing up public data is broader than just the space community.

      I work regularly with NASA's Planetary Data System, or PDS. It's a massive (~3.5petabytes!!) archive of off-world scientific data (largely but not all imaging data). PDS is integral for scientific research - public and private - around the world, and is maintained, for free, by NASA (with support of a number of Academic institutions).

      The current state of affairs for NASA is grim:

      And as a result, I (and many of my industry friends) have become increasingly concerned that PDS will be taken down as NASA is increasingly torn down for spare parts and irreparably damaged. This administration seems bent on destroying all forms of recording-keeping and public science, so who knows how long PDS will be kept up. Once it's down, it'll be a nightmare to try and collect it all again from various sources. I suspect we'll permanently lose decades worth of data - PDS includes information going all the way back to the Apollo missions!

      As such, we've been pushing to back-up as much of PDS as we can, but have absolutely no hope of downloading it all within the next year or two, nevermind in a few months if the current cuts impact us soon.

      If you or someone you know would be interested in helping figure out how we can back-up PDS before it's too late, please let me know here or in a DM. I've already tried reaching out to the Internet Archive, but did not hear anything back from them.

      Edit: to clarify, the larger problem is download speeds - we've topped out at 20mb/s with 8 connections.

      61 votes
    2. Does anyone have experience with tools for locally archiving the web, like Archivebox for example?

      I found myself on the Archivebox website earlier today. After reading some of it, that's the kind of program I could use. The ephemerous nature of the web is bothersome, so much content is lost...

      I found myself on the Archivebox website earlier today. After reading some of it, that's the kind of program I could use. The ephemerous nature of the web is bothersome, so much content is lost for one reason or another. Archivebox seems to be one of the most popular tools, and it can automatically mirror my locally downloaded website to archive.org, which is great. It seems complex though, maybe more complex than I usually tolerate these days. Which is why I am asking if anyone has personal experience with Archivebox or other similar programs. Do you find them useful and reliable? Have you ever found in your local storage a webpage that you really liked, which was gone from the web? How's your setup?

      Thank ;)

      19 votes