7
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I've been setting up my home network! It's been a heck of an experience, but things are working well now and I'm quite happy with it.
Physical Layer/Network Layer
Having recently broken Zillenial stereotypes by purchasing a home, I was determined to have good, professional network access throughout, especially since everyone living here works from home most of the time. Unfortunately, the house isn't wired for Ethernet, and the lowest estimate I got was $500/room, not including fixing the drywall and repainting.
I lucked out, though, because my ISP's provided modem didn't work when I plugged it into the only coax jack in my room, and they had to send a technician out. He was absolutely lovely, and we walked around the house together with his signal tester and discovered that the whole house is wired for coax in a very MoCA-friendly way; every bedroom has a coax jack, and there's one in the living room as well.
Currently, I have everything in my room, with a Ubiquiti UniFi U6-Lite "flying saucer" neatly mounted to the ceiling, powered by PoE through a simple vertical Ethernet line that comes out right next to the coax jack. This is the first time I've ever had a pure AP set up, and it's really damn nice; even when the router is completely screwed up, I can wirelessly connect and do maintenance (via WinBox in WINE) as if I were plugged in over Ethernet.
I plan to set up MoCA to every room that I can and put U6-Lites in all of them, so we have distinct and fast coverage everywhere. In addition, I got very lucky because my room is on the main face of the building, which faces towards a large park, and we are on the fifth floor. My 2.4 GHz network extends into a good portion of the park, so once it's no longer so bitterly cold, I'll be able to work from there.
Internetwork Layer
Because our building is supplied by Astound (neé RCN), I absolutely do not trust that the network will stay up consistently; they've been awful every time I've used them before. There happens to be a 5G cell tower about 200 feet away, and I signed up for T-Mobile's Home Internet plan - true unlimited 5G connectivity for $50 a month. I'm getting about 400Mbps consistently, comparable to the speed RCN provides over cable.
Currently, I have a single Mikrotik hEX S with the RCN connection coming into the SFP+ port and the T-Mobile connection coming in to
eth1. I haven't been able to get load balancing working well, so right now they're just in failover order (if the RCN network goes down, we switch all traffic to T-Mobile), but I'm thinking of setting up my older RouterBoard on the RCN network, getting an AS number, and running BGP between the two.Application Layer
In terms of actual applications, most of my network stuff is set up on my AS3304T 4-bay NAS. It's running four IronWolf drives in RAID 6, giving me 7 TB of storage and a two-drive failure tolerance. I currently just have SyncThing set up there, but I'm also looking at enabling Time Machine for the one Mac user in the house and automated rsync backups for me and my boyfriend.
In a few days, I'll be able to move my small Raspberry Pi cluster into the house, along with my FlightRadar24 feeder station. Being on the 5th floor, and much closer to the city center, will really help with my fr24 scores, or at least so I hope!
Overview
Basically, running Ethernet turns out to be a gigantic pain. Either you run it under the floor and destroy your floorboards, or your run it through the ceiling and have to drill a dozen joists per twenty feet. MoCA has gotten really good, and with a little elbow grease, a few electrician visits, and some luck, I think I'll be able to have a professional, usable networking setup without visible cables ruining my nicely painted (lavender, with a purple accent wall, at least in my room) walls.
Congrats on your new home!
We are mostly using WiFi but I did get ethernet to my desktop computer. There is just one wall between, so there is an ethernet jack on both sides and a very short cable directly connecting them in the wall, and ethernet cables running behind the couch to the router on one side and to my iMac on the other.
I was hoping I’d notice some difference in latency, but there was none at all. It seems WiFi is pretty good, in a small house, anyway. Maybe I’d notice if we got fiber, which is a possibility.
Thank you!
Being the IT person in the house, I have all my stuff connected by Ethernet; the need for WiFi is mostly for our smart home equipment (most of which we didn't buy; I'm basically allergic to that kind of thing, I hate it as a concept) and for WFH video calls from the others.
It's my experience that for most people the latency from whatever garbage their ISP uses for last-mile overwhelms that from WiFi as long as you don't need to use repeaters or mesh WiFi. Once you introduce multi-hop wireless at layer 2, you're basically screwed for anything <100ms. If you have a small house, a single AP is probably fine!
In our case, the real issue is that there are a lot of walls between the one existing AP and some of the rooms that really need connectivity, so while the latency is okay, the bandwidth is abysmal. That's not really workable for WFH video calling.
If Ethernet over coax works for you I wouldn't bother but you may be able to run cat6 simply by pulling the coax (but it may be stapled or not easily pulled). If not these drillbits combined with a fiber glass rod are great for getting cat6/cat5e anywhere you need it to go. I think I've read ethernet over coax caps out at around 1Gbps, shared across the entire network. Cat6 can do 10Gbps per connection with a decent switch.
Yep, MoCA caps out at 915Mbps, tragically. I suggested pulling unterminated cat6 over the existing coax runs but according to the electrician the coax is stapled to the beams, so, probably wouldn't work. I appreciate the advice though, if I ever get really fed up with this setup I will definitely spend some DIY time on this!
Yea I just went through the exact same thing in my house, there is a central brick wall that made WiFi off a central access point abysmal so I was looking into wireless mesh networks or alternative solutions. Ended up just running cat6 to a couple of additional access points and it was easier than I thought but that will vary widely depending on your situation.
How did you structure your AP networks? I'm not sure if this is ideal but I ended up giving them all identical SSIDs and passwords and letting the devices figure out which access point to talk to. You do get a brief hiccup if you're walking across my home with a device but otherwise it works quite well.
Yep, generally I think that's the way to go; devices running recent operating systems (like, since 2015) generally do a pretty good job of prioritizing the locally best BSSID/infrastructure station for a given SSID. The UniFi APs also do a "roaming nudge" to clients that aren't switching when they should by using the beamforming features of their transmitter to artificially lower the power.
I've implemented a binary classifier based on the paper Sketching Linear Classifiers over Data Streams in Rust.
The program is a commandline tool that allows you to discriminate between two classes using logistic regression. The interesting bit is that, due to the clever data structure described in the paper, the classifier uses very little memory (about 32 KiB) while still having reasonable accuracy (98.35% on the TREC2007 email spam/ham corpus when using Orthogonal Sparse Bigrams (PDF) as features). You can give the classifier more memory, which results in increased accuracy (e.g. 99.2% with 16MiB); this isn't state-of-the-art, but it is pretty good, and I'm really happy that it works as well as it does with limited memory.
However, to go beyond spam filtering as a use-case, I now need to figure out if I can somehow leverage the code for multi-label classification instead of binary classification (we need this at work). There are some methods that convert a set of binary classifiers into a multi-label one, but I haven't done enough research yet to settle on one. For now I'm using one classifier per label (presence vs. absence), but that probably won't scale well even with the small memory footprint, and it assumes that all labels are independent, which most definitively isn't true.
Eventually I would like to write a microservice that allows you to do arbitrary text classification: you give it an API key, a piece of text and the possible labels, and it'll figure out which ones apply to your given input after it sees enough training examples. As a pie-in-the-sky thought, it could theoretically even work as a public service for spam filtering (or general text classification) -- you can do Feature hashing in a way that makes it hard for spammers to corrupt your classifier unless they are in the majority.
This is really neat! With that little memory, this could potentially be really useful in inline network appliances; I wonder if anyone has thought to apply it to the WAF use case through e.g. Fastly Compute@Edge?
There are various so-called sketch data structures that approximate larger data structures. They are also useful in many networking use-cases (traffic monitoring, etc.).
The most-famous sketch is probably the bloom filter, but there's also the count-min sketch (which allows you to estimate frequencies of items in a data stream -- the linked paper cleverly uses that to store the classifier weights instead). Another fascinating application area is caching, where, for example, the Window Tiny-LFU sketch allows you to better utilize your cache memory by keeping items out of the cache that probably won't be requested again.
Neat, thanks for all the links - I've got a lot of reading to do :)
On the accordion MIDI controller project, I switched to making a harness that lets me wear a 25-key MIDI controller that I bought so it can be played like the right side of an accordion. There’s no practical reason to do this, but since an accordion is an instrument that you wear, it seemed about time to learn how.
Now I’m working on a centrifugal brake. This is a spinning wheel within a drum where the centrifugal force causes flyweights to rub against the inside of the drum. In theory this should cause braking proportional to the square of the speed that the wheel spins. (Had to review my high school physics for that.)
There are fishing reels that use this kind of braking, and it’s sometimes used for safety to prevent overspeed on industrial equipment. (Consider a crane that loses power.) I decided to try it first since it’s purely mechanical.
I've been working on a program that would effectively allow a massive reduction in needed hard drive space to archive lots of files. As a consequence of that, I've recently gone into a file format rabbit hole, looking into using something current that would suit my needs or designing my own. I haven't found much that can be reused, though.
Boy, format design is HARD. Every time I think I have something good or efficient, I think "What if X wasn't a hard requirement" or "What if Y is a valid use case", which makes me completely rethink what I've already thought of and slowly drives me madder. I'm trying to convince myself to settle on using a SQLite database as the container for a proof-of-concept program, but it makes me a bit sad considering that it doesn't have a lot of the properties that I initially wanted. I'm currently doing a small write-up of hard and soft requirements that will hopefully help me out a bit, since this has been mostly in my head. May post it to the Q&A thread if I'm stumped.
Not to mention that it's hard to search for/think of file extensions that won't clash horribly with something else. I think I can just pick, but I really don't feel comfortable using any particular extension. If I get over it, I may use
.lis,.las, or.mia, but I'm still thinking about it.Cache invalidation is relatively easy, IMO.
This feels like a good start for lossless compression...
Why not use a file system like ZFS or btrfs with zstd compression enabled for all files? ZFS can even do some fairly hefty data deduplication if I’m remembering correctly. Am I misunderstanding the purpose of your program?
I looked into it near the beginning. One of the hard requirements I have is that the result has to be easily distributable and usable for non-techies. Think something along the lines of competing with .zip or .rar files, and not uploading things to a NAS for browsable storage.
Even if I was to get ZFS working standalone and on files, ZFS' data deduplication works mostly on large identical data. My main use case is basically image files, and two files one would consider "identical" in terms of output pixels can have different representations on disk, so that wouldn't help much.
This is a frontrunner, but has some problems by itself which I'm trying to work around. A wrapper around
.tar.zstseems likely for the final format despite some issues with it.Are you planning on doing anything like lossy compression on the images? For example, the data in a png that had been changed from a bit depth of 8 to 16 is the same, but that’s difficult to check for - so identical output is a bit hard to deal with?
I'm trying to make it as lossless as possible, at the expense of initial conversion time. One problem I see is that a lot of the source images I'm going to be using are JPGs (high quality JPGs, but JPGs nonetheless), which will probably make me rethink a bunch of stuff. The only intentionally lossy things I may be doing is stripping metadata, and even then, I'm still thinking about if I should do that. (What happens after extracting the image data is going to be the user's choice, from lossless stuff to a "good enough" JPG.)
What do you mean by this? Unless I'm missing something, checking if an 8 bit colour depth was upscaled to 16 bits seems like something that would be easy to check for.
I'm not sure how well-known this is, but Dropbox's Lepton format can losslessly compress JPEGs by, as they claim, an average of 22%.
JPEG XL also has a mode that converts old JPEGs into the new format losslessly, with similar 20-30% savings in my experience.
Unfortunately neither of those formats is well-supported currently, but it might still make sense to use them internally.
I was aware of JPEG XL, but this is the first I'm hearing of Lepton. I'll try it out for comparison once I get some other things up and running, thanks a lot.
I built a Corne keyboard over the weekend--my third one, but first wireless build using nice!nano controllers instead of Elite Cs, and first choc build using kailh low-profile switches and a flat keycap profile.
Been spending a bit of time in zmk, trying to get my keymap to match the one I use in qmk on the other two boards, which has been a challenge because I use both mouse keys and macros, which aren't yet officially supported in zmk. There's an existing PR that adds mouse keys, which is working fine for me, and I managed to find a work-around for my one missing macro by moving the functionality into a shell script that I invoke instead of having the keyboards play it out.
Overall I'm happy with the new wireless board. The vendor I bought from offers a sleek aluminum case which was tempting, but my wife is already balking at how much I'm spending on this keyboard-building hobby so I decided to dust off the 3D printer and make my own case instead, which turned out nicer than I was expecting.
I put a photo of all three boards here.
I found this amazing tool for applying an sRGB clamp to wide gamut monitors on Nvidia GPUs. It is really useful if you have a high end monitor that lacks a proper sRGB emulation mode despite good factory calibration (like mine).
I forked it with the goal of adding some new features, such as automatically turning the clamp off when in HDR mode, and re-applying the clamp whenever the display wakes.
I made a floating window manager for i3. Here it is in action: https://i.imgur.com/IVnZ9fS.mp4