• Activity
  • Votes
  • Comments
  • New
  • All activity
  • Showing only topics in ~tech with the tag "data". Back to normal view / Search all groups
    1. Not sure if coincidence or I should give up (on USB flash drives)

      Hey Tilderinos. I've been looking into buying several flash drives since my largest flash drive is a 32GB sandisk, and I use or interact with all the 3 major OSes, I use Linux on my desktop and on...

      Hey Tilderinos.
      I've been looking into buying several flash drives since my largest flash drive is a 32GB sandisk, and I use or interact with all the 3 major OSes, I use Linux on my desktop and on a secondary laptop, I use MacOS on a Macbook and everyone else I know uses Windows(So I'll need an exFAT drive for them).
      My recent experience with flash drives though makes me more willing to trust my data to a system's RAM than to a flash drive. At least RAM wouldn't lure me into a false sense of security then spontaneously fail, I know that my data isn't going to last a reboot.
      I've got 3 sandisk cruzer blades fail on me, once was an error on my part where I accidentally hit it with my knee while plugged into a device(device unharmed, the drive is dead), one time I upgraded the SSD on my SteamDeck and flashed the steamdeck recovery to an 8GB stick, it worked fine while restoring and it still can be read... it's stuck on read only and, Gparted, Windows formatter, Rufus, Mac's disk utility nor mkfs can make it reusable, I assume it entered read only because it tripped some "whoops I'm dying" thing like some SSDs have(from what I know). The last one is effectively a resistor that connects to a USB port, it heats up, SOMETIMES appears on PC, Linux can open it, copying things into it via Nautilus works albeit very slowly, then when I try to open the root of the flash drive it is stuck perpetually trying to load, when I unplug it and plug it back in again, I can see the folders but entering any of them immediately goes into the permanent loading state, mkdir, cd and ls can work on the drive... intermittently, but I'm treating it as dead.

      This leaves me with only 2 other drives, my largest drive, is a 32GB Sandisk Ultra, and my smallest drive which has been more reliable than the Cruzer Blades is a 4GB Sony... USM4GP thing, which I have no idea what year it was from and a quick googling didn't bring up any release date, but it had faster read/write than the cruzer blades, and it's been my main drive for things like installing an OS/burning an image into it yet it lasted all these years.

      I'm trying to get a USB-C drive specifically since all my laptops that are in use and my PC have a USB-C port, but all I see in local big retailers are Sandisk, Sandisk and more Sandisk, Amazon however has some that aren't Sandisk.

      Does anyone know a USB-C flash drive that is genuinely reliable? Was it specifically Cruzer Blades that is garbage? I've not had a Sandisk SD card fail on me yet, should I just avoid Sandisk for anything but SD cards? Should I just say fuck it and buy a bunch of enclosures and NVMe drives?

      I've read that flash drives get bottom of the barrel NAND chips that can't be used on SSDs, too. I know that flash drives aren't meant for long term storage/backup but a drive that old shouldn't be still going on that strongly against new ones.

      I've been looking at PNY Elite V3 with USB-C as a connector that I've seen a few listings on Amazon but it's 1TB price is almost the same price as a 1TB NVMe SSD(Though not factoring in the enclosure).

      25 votes
    2. Looking for feedback on a homelab design

      I wanted some help with a homelab server I am in the beginning stages of designing. I am looking for a flexible and scalable media and cloud system for home use, and I thought this community would...

      I wanted some help with a homelab server I am in the beginning stages of designing. I am looking for a flexible and scalable media and cloud system for home use, and I thought this community would be a good place to source feedback and recommendations before taking any real next steps! I really want to check that I am approaching the architecture correctly and not making any bad assumptions. I am open to all feedback, so please let me know what you think!
      I already run a simple home server and I have typical homelab FOSS apps, such as jellyfin, navidrome and audiobookshelf, but I am also interested in migrating away from cloud storage using nextcloud, immich, etc. In an ideal world, this setup would also allow me to leave windows on my main machine and use a windows vm for business related work that can’t be done on Linux. I will likely be the one primarily using the services, however I could expect up to 10 - 20 users eventually.

      High level setup is with two machines:

      • Proxmox Server
      • TrueNAS Scale server
        • JBOD with either 90 bay or 45 bay storage
      • 10G switch

      This might be a stupid setup right off the bat, which is why I wanted to discuss it with you all! I have read a ton about using TrueNAS as a WM within Proxmox, but I just like the idea of different machines handling different tasks. The idea here would be to set up the TrueNAS server so it can be optimized for managing the storage pool to allow for easy growth. While the Proxmox server can handle all the VMs and connecting users, with higher IO, etc.

      TrueNAS System Specs:

      • AMD ryzen CPU and motherboard
      • 64 or 128GB ram
      • Mirror 500GB M.2 NVMe OS Drives
      • GPU if necessary, but hopefully not needed
      • Dual 10gb pcie card if the motherboard doesnt already come with them
      • An hba for the JBOD something like the LSI SAS 9305-16e
      • SLOG and L2ARC as necessary?

      JBOD enclosure

      • While I am interested in a 90-bay enclosure, I would only realistically be starting with two vdevs which is why I think a 45 bay enclosure wouldn’t be an issue.
      • Im tentatively planning for an 11 wide Raidz2 vdev configuration. This would hopefully scale to 8 vdevs with 2 hot spares or 4 vdevs with 1 hot spare.
      • All drives would be HDDs

      Proxmox Server Specs:

      I am less familiar with the specs I will need for a good Proxmox server, but here is what I am thinking.

      • AMD epyc and motherboard if I can get my hands on a less expensive one. Otherwise I was thinking a higher end AMD ryzen cpu
      • 128 or 256GB ram
      • Mirror 500GB M.2 NVMe OS Drives
      • Somewhere between 2 and 8 TBs of SSD storage. Depending on the number of drives, I think this would be a single drive, mirror or raidz1.
        • This storage will be used for all the vm configuration and storage, except for something like Nextcloud where the main storage will go onto the TrueNAS mount.
        • I would also use this for temporal storage such as downloading a file before transferring it to the TrueNAS mount.
      • A dedicated GPU primarily for transcoding media streams, but also for testing and experimenting with different AI models.
      • Dual 10gb pcie card

      Questions:

      • I know Proxmox can do zfs right out of the box so I know I don’t need the TrueNAS server, but splitting it this way just seems more flexible. Is this a realistic setup or would it just be better to let Proxmox do everything?
        • Does anyone have experience creating NFS shares in TrueNAS for mounting in Proxmox? I would be interested in thoughts on performance, and stability among any other insights.
      • Do any of the system specs I listed seem out of line? Where and how do you think things should be scaled up or down?
      • If I ever did expand to a second JBOD shelf, assuming the first one was full first, is it be possible to create new vdevs that spanned across the shelfs without losing data?
      • Is SLOG and/or L2ARC necessary for this setup? What capacity and configuration would be best?
      • What else have I missed?

      Lastly, a quick blurb:

      I have been building PCs for a while and undertook building a home server a few years ago. I loved the experience of learning Linux (the server is running Ubuntu), picking up docker, and learning more about the FOSS community has been a joy! Part of this project is to learn along the way but also have a setup that I can build towards over time! Proxmox, TrueNAS and zfs would all be new to me so I really see it as an opportunity to explore. I want a solid media and cloud server setup, while also giving myself the freedom to explore new operating systems and general hypervisor functionality.

      22 votes
    3. Photo digitizing

      Hi all, I've got (probably) a few thousand family photographs that I plan on scanning/digitizing. These photographs are organized into dozens or hundreds of envelopes with month/year and sometimes...

      Hi all,

      I've got (probably) a few thousand family photographs that I plan on scanning/digitizing. These photographs are organized into dozens or hundreds of envelopes with month/year and sometimes event description written on them. I'm on the fence between using a service to do it or DIYing it with a scanning machine.

      The way I see it is -

      Service pros:

      1. I don't have to do it myself

      Service cons:

      1. I may have no control over how the digitized photos are tagged or organized (date tagged, filename)
      2. Risk of photographs being lost/damaged
      3. $$$$

      DIY pros:

      1. I can tag and organize the photos exactly how I want
      2. Much less expensive

      DIY cons:

      1. I have little spare time and this project could be extremely time consuming.

      I would love to hear if anyone here has experience doing this and what techniques or pitfalls you may have discovered along the way.

      7 votes
    4. I hate the new internet. I hate the new tech world. I hate it all. I want out, and I can't be the only one.

      I think most people would agree that the internet and technology in general have absolutely gone to shit over the past decade or so. There is no corner of the internet nor of the software world...

      I think most people would agree that the internet and technology in general have absolutely gone to shit over the past decade or so. There is no corner of the internet nor of the software world that hasn't been affected by enshittification. Everything exists to serve you ads. Everyone wants to extract as much money from you as possible. Every website is in a race for the bottom as they try to find the lowest effort content that makes them the most money. Every piece of software is pushed out half-baked and/or stripped down to the bare minimum with the rest paywalled or with the devs pinky promising to fix it 5 updates down the road.

      Every social medium is just bots. The front page of Reddit is easily 35% easily detectable bots at least and who knows what the rest is comprised of. And it's probably the one that's doing the best at the moment, Facebook, Twitter, Instagram, Tiktok, all of them are just bots and propaganda and engagement farming the whole way down. And the worst thing is, they're complicit. Hell, they're actively encouraging it and trying to find ways to make it worse. And I have no doubt Reddit will bend the knee soon enough too (they just banned /r/whitepeopletwitter because Musk made a tweet critical of the sub).

      There's probably some element of rose-tinted glasses here, but the old internet was just so much better looking back. Like, early 2000's to maybe 2012, 2013 or so, that was the peak. No colossal data harvesting schemes feeding into algorithms designed to keep you engaged on their site 24/7 for the purpose of shilling you advertisements and selling your data, no mass propaganda, no Dead Internet Theory (which can hardly be considered a theory anymore). Yeah there was shit content, there was tons of it, but I can deal with shit content and petty forum drama and whatnot; what I can't deal with is all the multi-billion dollar corporations trying to shape the entire landscape of the Web into the perfectly minmaxxed cash-generating machine that does as little as possible for as much data and advertising as possible.

      Modern software isn't much better. Windows and MacOS are filled with anti-user features, telemetry you just can't turn off, Windows will often just install shit on your computer without telling you. They turn your computer into a walled garden, where you can do what you want as long as you play by their rules, but without giving you any real control over what your computer does. Yeah you can delete system files and brick your laptop if you feel like it, but anyone who's ever tried to permanently disable Windows updates will know that in the end you're not the one calling the shots: Microsoft are. And... Like, that's insane, right? It's running on my fucking computer, it's my CPU doing the work, I want to know what the hell it's doing and not just the parts it lets me see, and if I want it to do something different then I should be able to make it so.

      I hate it all. I'm tired. I want out.


      These are my problems. Here's what I've done about it so far.

      • Obsessive privacy on the web. No Google services. Firefox with as much telemetry turned off as possible. Protonmail and ProtonVPN for everything (and I'm considering getting out of those too with the pro-Trump stances they've been taking recently). As minimal an online footprint as I can get, I make as few accounts as possible and I don't use shared or even slightly related usernames (my username here is an exception as it's my Reddit username, and no, it's not my real name), I delete accounts whenever I can and I GDPR request the services afterward. Virtual cards for online payments as much as possible. Will probably make a Javascript whitelist at some point too. Is all of this overkill? Yes. Why do I bother? Because fuck them.

      • As little social media presence as possible. Real life necessitates some amount of social media interaction of course, I have Facebook and Instagram but use them exclusively for messaging. I often see people excluding Reddit from social media but I don't fully agree, even if it's not exactly in the category it still targets a lot of the same psychological weak points in us, encouraging doom scrolling and shaping our opinions through echo chambers and propaganda (it's always important to remember that echo chambers and propaganda you agree with are still echo chambers and propaganda). I still use Reddit admittedly, but I've tried to minimise my usage as much as possible and I'm shopping for alternatives.

      • Free and Open Source software as much as possible. I'm all in on GNU these days. Yes, it's a massive pain in the ass. My job unfortunately requires some Windows-only software so I'm running a dual partition but I'm trying to get as much of my computer usage onto Linux as possible (I use Arch btw). Like I said above, it's my computer, if I can't control what it's computing then it stops being my computer, it's at best shared between me and all the developers of the proprietary software I have installed on it.


      That's my rant. It's been a long time coming.

      There are still things I'm looking to change, especially with how I use the internet. Getting rid of Reddit is the next big step for me, I think. I just can't be bothered with it anymore, but there is still something about it that I love, every time I look through a small niche topic community, or an interesting new hobby sub I've never seen before with years of cool posts for me to go through. And yeah, I do still enjoy browsing through /r/all even when it's 80% shit and objectively bad for my mental health. But at this point the overwhelming mass of utter shit is just not worth digging through anymore. I'm tired.

      Tildes is really cool. It reminds me of the old internet, the ideal usage of the Web. I open the site, I see a link to an interesting article, I read it, I give it a like, I read and/or contribute to the discussion in a comments section. I want more of this.

      If anyone has any links to cool sites that I should check out I'd greatly appreciate it.

      165 votes
    5. I have no idea to advance in my career toward data science

      I did a masters in data analytics, and then the niche I fell into in the working world was building dashboards, reports and spreadsheets of financial data for non-technical bureaucrats. Instead of...

      I did a masters in data analytics, and then the niche I fell into in the working world was building dashboards, reports and spreadsheets of financial data for non-technical bureaucrats. Instead of ensuring data quality by technical means, my current company often just has me manually reviewing and checking financial data. This is pretty frustrating to me because I have no education in finance, and the things I miss or get wrong are so second nature to my boss that he doesn't even see them as something I should have been trained on. The only technologies I use are SQL server and excel. Any proactive steps I've made to automate processes has been discouraged as not worth the time.

      I'm aware that most people spend years on tedious stuff before ever getting to work with more engaging technology, but honestly I'm starting to wonder if they've forgotten I'm not a finance guy. I want to move up in my career especially to escape my current role, but I'm feeling completely lost as to how. There's no obvious role in my company that could be a 'next rung of the ladder' to advance into, so there's nobody I can emulate to help chart a course. My boss had an unconventional path to his current role, and isn't really into manager stuff like career mentoring, so he's no help in that regard.

      To anyone with experience in data science, what is the advancement supposed to look like? What are the key skills I should be developing? Am I being too averse to learning the subject matter of the data I'm working on? Any insight is appreciated!

      13 votes
    6. eBay privacy policy update and AI opt-out

      eBay is updating its privacy policy, effective next month (2025-04-27). The major change is a new section about AI processing, accompanied by a new user setting with an opt-out checkbox for having...

      eBay is updating its privacy policy, effective next month (2025-04-27). The major change is a new section about AI processing, accompanied by a new user setting with an opt-out checkbox for having your personal data feed their models.

      While that page specifically references European areas, the privacy selection appears to be active and remembered between visits for non-Europe customers. It may not do anything for us at all. On the other hand, it seems nearly impossible to find that page from within account settings, so I thought I'd post a direct link.

      I'm well aware that I'm anomalous for having read this to begin with, much less diffed it against the previous version. But since I already know that I'm weird, and this wouldn't be much of a discussion post without questions:

      • How do you stay up to date with contract changes that might affect you, outside of widespread Internet outrage (such as recent Firefox news)?
      • What's your threshold -- if any -- for deciding whether to quit a company over contract changes? Alternatively, have you ever walked away from a purchase, service, or other acquisition over the terms of the contracts?
      46 votes
    7. Experience with data protection laws (GDPR, ePD, CCPA, etc..)

      This is a topic I keep revisiting. It's constantly evolving, with new laws in different parts of the world happening pretty often. And also there's a lot of grey area with vague or incomprehensive...

      This is a topic I keep revisiting. It's constantly evolving, with new laws in different parts of the world happening pretty often. And also there's a lot of grey area with vague or incomprehensive language that hasn't yet been tested in courts.

      I recognize that it's a bit of a niche topic, but I think there are a lot of us at Tildes who have to think about it. After all it potentially impacts anyone maintaining or building a non-platform web presence. It also applies to less obvious things like running an advertising campaign that involves media requested from a server you control (which can therefore potentially log requests).

      For my part, I've needed to research laws relating to PII in order to come up with policies and practices in various contexts. In broad strokes it's pretty simple but as you get into details what I continue to find is that there are a lot of conflicting opinions both from professionals and lawyers. A lot of it is still open to interpretation.

      I'm wondering what kinds of experience other tildenauts have around data protection and PII? Have you implemented solutions? Do you wonder about it for your own websites? Have you been involved with it at companies where you've worked? Do you have questions about it?

      13 votes