7
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I have, for a few months now, been writing a feed reader web service: Russet. Mostly to scratch an itch, but also to give myself some Rust web service experience and build up an up-to-date portfolio item.
I'm pretty happy with it so far! I've been dogfooding it for a few months now, and I would definitely call it a "resounding success" on all intended fronts (even though one of those fronts is "fodder for complaints about async Rust", which it has provided in spades, lol… I'll dig into that another time, maybe).
I’ve gotten distracted from working on Keeper (my link-sharing forum project) by issues around how to do property-based testing. Fast-check’s API isn’t as flexible as I’d like; implementing your own Arbitrary from others using map(), chain(), and filter() is traditional (it’s how QuickCheck works), but I don’t particularly like it compared to generating one value at a time in a generate() function, and implementing that by subclassing Arbitrary isn’t great either, due to having to implement shrinking yourself.
While reading the fast-check source code, I found a bug in how fast-check handles streams. For a while I thought a splittable random number generator was needed, so that got me reading about random number generators. I even ported one to JavaScript, before deciding it wasn’t quite what I wanted.
It’s a pleasant distraction. It’s fun reading random number generator papers and seeing all the different approaches people have tried.
I’m also looking at how property-based testing is done in other languages. In Python, Hypothesis takes quite a different approach, which is also used in Rust. These libraries are closer to fuzzers; shrinking is done by changing the “random” data consumed by the code that generates Arbitraries. (Which isn’t really random anymore; it’s just a stream of bytes.)
My plan now is to build a toy property-based testing library and then think about what improvement might be made to fast-check.
Been bouncing between hobbies lately, so work on Settlemint (my expense tracking app project) slowed down a little.
However I did make some more progress cleaning up the code and trimming unnecessary fat, and now I feel like I'm almost done so I can start work on features again.
I'm unsure about how well I'm handling login though. When a user is created, the frontend will send a SHA256 hash of the password to the server, which will hash and salt it using Argon2.
When logging in, I also SHA256 the password on the frontend then send it to the server for validation.
I'm unsure if this is sufficient though. I could do asymmetric encryption of the login information, but I'm unsure if it's overkill.
Has anyone worked on login functionality for a personal app or at work? What approach did you use?
If you're using HTTPS for transport, there's not much benefit in hashing the password clientside. If you inspect the network calls for login on sites that you use (including this very one), you'll see that the password is sent as plaintext.
Argon2 is a good choice for hashing it prior to storing in the DB.
My dev server is using HTTP for now, but I will certainly use HTTPS for all production deployments.
Yeah, I knew hashing the password client-side would be redundant but I did it anyway, guess I'll remove the hashing function call and just send it over as plaintext.
Inspired by the recent thread about getting someone into a hobby with a $250 budget (IIRC), I decided to finally go forward with the Homelab concept. I had known about it for years, but never really felt enough desire to implement it. Then the thread changed my mind.
On a humorous note, it will be slightly verbally confusing if I talk to others about my hobbies, since I already have a home electronics lab.
I found a scrap/ewaste reseller on ebay who had 3 working (though scratched up) Dell Precision 7910 workstations with high-end specs for the time: 2x e5-2690 v4 Xeon 14-core processors, 128GB ECC DDR4, quad-output Nvidia Quadro M2000 cards, on-board capacity for 8 SAS drives with built-in RAID functionality, dual on-board gigabit ethernet ports, USB 3.0, SEVEN total PCI slots (6 various PCIe and 1 old-school PCI). These are 7 year old systems that would have been priced at somewhere over the $10,000 mark when they were made. The chassis is a monster and the whole tower is really well designed. This gives me a base of 28 cores and 128GB RAM to start with, and the option to go up to 44 cores (dual 2699's) and up to 1TB of RAM.
I bid on and won 2 of them for right around $600 total. $300/ea for those workstations, with those specs was well below current market rate and I'm happy with the purchase. They arrived a couple days ago and I've been looking them over before buying other stuff: going to get some SAS drives for NAS functionality and a PCIe M.2 NVMe adapter card so I can run a NVMe drive as the system drive. Considering putting in Noctua coolers, but I'll start off with getting some services up and running and then decide based on running temps if that is called for. Also interested in seeing what all those cores will do for me if I feel like doing some video editing.
As for the actual services I want to implement? Custom DHCP along the lines of Pi-Hole for fully network-implemented ad-blocking. NAS for easier, centralized storage to both access and move files between the computers we have. Media service (Plex / Jellyfin / other?) to have a nice UX to access the media on the NAS so we have offline network TV (we tend to sometimes have internet outages, but very rarely do we get a power outage where we are, underground utilities and newer infrastructure). Oh, and I'll probably run Proxmox as my layer on bare metal so I can get experience using a hypervisor, as well as Docker to run my services in containers.
Maybe later I'll pick up a cheap Pascal-generation Nvidia Tesla compute card to see about training/running AI models locally. Try to play around with AI during the brief window before it goes Skynet and decides humanity needs to go.
So, that's where I'm planning on starting and I'll see what I'd like to expand to as I go.
Those are some beefy machines with the power draw to go with it. Curious why you bought two?
For your current use-case, it seems like overkill to another level. That said, I'd be giddy to toy around with them myself.
I'm a big believer in redundancy. If I end up implementing services we come to rely on, having an unused, safely not-plugged-in identical platform ready on-hand seems excellent.
For the services I'm going to start with? Absolutely overkill. That said, I am also a big believer in overkill. Maxim 37: There is no "overkill." There is only "open fire" and "reload." Also, if I want to use the hardware for video editing / rendering and/or AI then it rapidly goes from 'overkill' to 'reasonably good' or 'barely adequate' (AI depending on GPU-architecture compute far more than CPU-architecture compute).
OS: I am planning on running Proxmox as my layer on bare metal. After that... hmm. I am currently most familiar with Linux Mint, but while it has been excellent for a desktop environment, and while I am confident I could make it work for server purposes, I'd probably be better served by a different distro. Considerations are currently Debian and TrueNAS, though I'm still in the research phase on what would be best for my use-case and very open to trying a shotgun approach for the learning experience.
I have no idea as of yet about implementing AI, but I know where the resources are for me to read up on it and it's not my first, second, or even third priority right now.
As part of my
mvyak-shaving I added more granular flags for controlling what happens when a file is trying to overwrite a folder and when a folder is trying to overwrite a file.Seeing this Tildes thread motivated me to try adding multi-threading again but 3 of the ~80 tests still fail when doing that so not enabling that for now... but I was pleased to see that file operation order no longer seems to matter so now it can start immediately rather than needing to first fully scan each source folder...
edit: when I leave the clobbering part a single thread then the moving part multi-thread then all the tests pass. I think that will have to do for now