7 votes

What programming/technical projects have you been working on?

This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?

24 comments

  1. [7]
    mtset
    Link
    I've been setting up my home network! It's been a heck of an experience, but things are working well now and I'm quite happy with it. Physical Layer/Network Layer Having recently broken Zillenial...

    I've been setting up my home network! It's been a heck of an experience, but things are working well now and I'm quite happy with it.

    Physical Layer/Network Layer

    Having recently broken Zillenial stereotypes by purchasing a home, I was determined to have good, professional network access throughout, especially since everyone living here works from home most of the time. Unfortunately, the house isn't wired for Ethernet, and the lowest estimate I got was $500/room, not including fixing the drywall and repainting.

    I lucked out, though, because my ISP's provided modem didn't work when I plugged it into the only coax jack in my room, and they had to send a technician out. He was absolutely lovely, and we walked around the house together with his signal tester and discovered that the whole house is wired for coax in a very MoCA-friendly way; every bedroom has a coax jack, and there's one in the living room as well.

    Currently, I have everything in my room, with a Ubiquiti UniFi U6-Lite "flying saucer" neatly mounted to the ceiling, powered by PoE through a simple vertical Ethernet line that comes out right next to the coax jack. This is the first time I've ever had a pure AP set up, and it's really damn nice; even when the router is completely screwed up, I can wirelessly connect and do maintenance (via WinBox in WINE) as if I were plugged in over Ethernet.

    I plan to set up MoCA to every room that I can and put U6-Lites in all of them, so we have distinct and fast coverage everywhere. In addition, I got very lucky because my room is on the main face of the building, which faces towards a large park, and we are on the fifth floor. My 2.4 GHz network extends into a good portion of the park, so once it's no longer so bitterly cold, I'll be able to work from there.

    Internetwork Layer

    Because our building is supplied by Astound (neé RCN), I absolutely do not trust that the network will stay up consistently; they've been awful every time I've used them before. There happens to be a 5G cell tower about 200 feet away, and I signed up for T-Mobile's Home Internet plan - true unlimited 5G connectivity for $50 a month. I'm getting about 400Mbps consistently, comparable to the speed RCN provides over cable.

    Currently, I have a single Mikrotik hEX S with the RCN connection coming into the SFP+ port and the T-Mobile connection coming in to eth1. I haven't been able to get load balancing working well, so right now they're just in failover order (if the RCN network goes down, we switch all traffic to T-Mobile), but I'm thinking of setting up my older RouterBoard on the RCN network, getting an AS number, and running BGP between the two.

    Application Layer

    In terms of actual applications, most of my network stuff is set up on my AS3304T 4-bay NAS. It's running four IronWolf drives in RAID 6, giving me 7 TB of storage and a two-drive failure tolerance. I currently just have SyncThing set up there, but I'm also looking at enabling Time Machine for the one Mac user in the house and automated rsync backups for me and my boyfriend.

    In a few days, I'll be able to move my small Raspberry Pi cluster into the house, along with my FlightRadar24 feeder station. Being on the 5th floor, and much closer to the city center, will really help with my fr24 scores, or at least so I hope!

    Overview

    Basically, running Ethernet turns out to be a gigantic pain. Either you run it under the floor and destroy your floorboards, or your run it through the ceiling and have to drill a dozen joists per twenty feet. MoCA has gotten really good, and with a little elbow grease, a few electrician visits, and some luck, I think I'll be able to have a professional, usable networking setup without visible cables ruining my nicely painted (lavender, with a purple accent wall, at least in my room) walls.

    7 votes
    1. [2]
      skybrian
      Link Parent
      Congrats on your new home! We are mostly using WiFi but I did get ethernet to my desktop computer. There is just one wall between, so there is an ethernet jack on both sides and a very short cable...

      Congrats on your new home!

      We are mostly using WiFi but I did get ethernet to my desktop computer. There is just one wall between, so there is an ethernet jack on both sides and a very short cable directly connecting them in the wall, and ethernet cables running behind the couch to the router on one side and to my iMac on the other.

      I was hoping I’d notice some difference in latency, but there was none at all. It seems WiFi is pretty good, in a small house, anyway. Maybe I’d notice if we got fiber, which is a possibility.

      2 votes
      1. mtset
        Link Parent
        Thank you! Being the IT person in the house, I have all my stuff connected by Ethernet; the need for WiFi is mostly for our smart home equipment (most of which we didn't buy; I'm basically...

        Thank you!

        Being the IT person in the house, I have all my stuff connected by Ethernet; the need for WiFi is mostly for our smart home equipment (most of which we didn't buy; I'm basically allergic to that kind of thing, I hate it as a concept) and for WFH video calls from the others.

        It's my experience that for most people the latency from whatever garbage their ISP uses for last-mile overwhelms that from WiFi as long as you don't need to use repeaters or mesh WiFi. Once you introduce multi-hop wireless at layer 2, you're basically screwed for anything <100ms. If you have a small house, a single AP is probably fine!

        In our case, the real issue is that there are a lot of walls between the one existing AP and some of the rooms that really need connectivity, so while the latency is okay, the bandwidth is abysmal. That's not really workable for WFH video calling.

        3 votes
    2. [4]
      streblo
      Link Parent
      If Ethernet over coax works for you I wouldn't bother but you may be able to run cat6 simply by pulling the coax (but it may be stapled or not easily pulled). If not these drillbits combined with...

      Unfortunately, the house isn't wired for Ethernet, and the lowest estimate I got was $500/room, not including fixing the drywall and repainting.

      If Ethernet over coax works for you I wouldn't bother but you may be able to run cat6 simply by pulling the coax (but it may be stapled or not easily pulled). If not these drillbits combined with a fiber glass rod are great for getting cat6/cat5e anywhere you need it to go. I think I've read ethernet over coax caps out at around 1Gbps, shared across the entire network. Cat6 can do 10Gbps per connection with a decent switch.

      2 votes
      1. [3]
        mtset
        Link Parent
        Yep, MoCA caps out at 915Mbps, tragically. I suggested pulling unterminated cat6 over the existing coax runs but according to the electrician the coax is stapled to the beams, so, probably...

        Yep, MoCA caps out at 915Mbps, tragically. I suggested pulling unterminated cat6 over the existing coax runs but according to the electrician the coax is stapled to the beams, so, probably wouldn't work. I appreciate the advice though, if I ever get really fed up with this setup I will definitely spend some DIY time on this!

        2 votes
        1. [2]
          streblo
          Link Parent
          Yea I just went through the exact same thing in my house, there is a central brick wall that made WiFi off a central access point abysmal so I was looking into wireless mesh networks or...

          Yea I just went through the exact same thing in my house, there is a central brick wall that made WiFi off a central access point abysmal so I was looking into wireless mesh networks or alternative solutions. Ended up just running cat6 to a couple of additional access points and it was easier than I thought but that will vary widely depending on your situation.

          How did you structure your AP networks? I'm not sure if this is ideal but I ended up giving them all identical SSIDs and passwords and letting the devices figure out which access point to talk to. You do get a brief hiccup if you're walking across my home with a device but otherwise it works quite well.

          1 vote
          1. mtset
            Link Parent
            Yep, generally I think that's the way to go; devices running recent operating systems (like, since 2015) generally do a pretty good job of prioritizing the locally best BSSID/infrastructure...

            Yep, generally I think that's the way to go; devices running recent operating systems (like, since 2015) generally do a pretty good job of prioritizing the locally best BSSID/infrastructure station for a given SSID. The UniFi APs also do a "roaming nudge" to clients that aren't switching when they should by using the beamforming features of their transmitter to artificially lower the power.

            2 votes
  2. [4]
    DataWraith
    Link
    I've implemented a binary classifier based on the paper Sketching Linear Classifiers over Data Streams in Rust. The program is a commandline tool that allows you to discriminate between two...

    I've implemented a binary classifier based on the paper Sketching Linear Classifiers over Data Streams in Rust.

    The program is a commandline tool that allows you to discriminate between two classes using logistic regression. The interesting bit is that, due to the clever data structure described in the paper, the classifier uses very little memory (about 32 KiB) while still having reasonable accuracy (98.35% on the TREC2007 email spam/ham corpus when using Orthogonal Sparse Bigrams (PDF) as features). You can give the classifier more memory, which results in increased accuracy (e.g. 99.2% with 16MiB); this isn't state-of-the-art, but it is pretty good, and I'm really happy that it works as well as it does with limited memory.

    However, to go beyond spam filtering as a use-case, I now need to figure out if I can somehow leverage the code for multi-label classification instead of binary classification (we need this at work). There are some methods that convert a set of binary classifiers into a multi-label one, but I haven't done enough research yet to settle on one. For now I'm using one classifier per label (presence vs. absence), but that probably won't scale well even with the small memory footprint, and it assumes that all labels are independent, which most definitively isn't true.

    Eventually I would like to write a microservice that allows you to do arbitrary text classification: you give it an API key, a piece of text and the possible labels, and it'll figure out which ones apply to your given input after it sees enough training examples. As a pie-in-the-sky thought, it could theoretically even work as a public service for spam filtering (or general text classification) -- you can do Feature hashing in a way that makes it hard for spammers to corrupt your classifier unless they are in the majority.

    5 votes
    1. [3]
      mtset
      Link Parent
      This is really neat! With that little memory, this could potentially be really useful in inline network appliances; I wonder if anyone has thought to apply it to the WAF use case through e.g....

      This is really neat! With that little memory, this could potentially be really useful in inline network appliances; I wonder if anyone has thought to apply it to the WAF use case through e.g. Fastly Compute@Edge?

      2 votes
      1. [2]
        DataWraith
        Link Parent
        There are various so-called sketch data structures that approximate larger data structures. They are also useful in many networking use-cases (traffic monitoring, etc.). The most-famous sketch is...

        There are various so-called sketch data structures that approximate larger data structures. They are also useful in many networking use-cases (traffic monitoring, etc.).

        The most-famous sketch is probably the bloom filter, but there's also the count-min sketch (which allows you to estimate frequencies of items in a data stream -- the linked paper cleverly uses that to store the classifier weights instead). Another fascinating application area is caching, where, for example, the Window Tiny-LFU sketch allows you to better utilize your cache memory by keeping items out of the cache that probably won't be requested again.

        1 vote
        1. mtset
          Link Parent
          Neat, thanks for all the links - I've got a lot of reading to do :)

          Neat, thanks for all the links - I've got a lot of reading to do :)

          1 vote
  3. skybrian
    Link
    On the accordion MIDI controller project, I switched to making a harness that lets me wear a 25-key MIDI controller that I bought so it can be played like the right side of an accordion. There’s...

    On the accordion MIDI controller project, I switched to making a harness that lets me wear a 25-key MIDI controller that I bought so it can be played like the right side of an accordion. There’s no practical reason to do this, but since an accordion is an instrument that you wear, it seemed about time to learn how.

    Now I’m working on a centrifugal brake. This is a spinning wheel within a drum where the centrifugal force causes flyweights to rub against the inside of the drum. In theory this should cause braking proportional to the square of the speed that the wheel spins. (Had to review my high school physics for that.)

    There are fishing reels that use this kind of braking, and it’s sometimes used for safety to prevent overspeed on industrial equipment. (Consider a crane that loses power.) I decided to try it first since it’s purely mechanical.

    3 votes
  4. Bauke
    (edited )
    Link
    Tildes ReExtended I finally got back to working on Tildes ReExtended and fixed up a few bugs, added 3 new features as well as the decided user labels editor (which you can access from the User...

    I finally got back to working on Tildes ReExtended and fixed up a few bugs, added 3 new features as well as the decided user labels editor (which you can access from the User Labels section). I also made it so it no longer opens automatically when the version is updated and instead added indicators to new features which I think looks pretty cool. Maybe I'll add another indicator for updates too.

    After getting back to TRX I also spent some time making a WebExtension template with all the stuff I want (HTM, Preact, webextension-polyfill, etc.) and hopefully it will be the base for a few other extensions I have in my head that I want to make.

    I don't think I've really said much about Queue but I think it's probably the best thing I've ever made. It's a WebExtension that adds an "Add to Queue" button to context menus for links and tabs, and if you have anything queued, clicking on the extension icon will send you to the next item. The entire functionality is described in the screenshot on AMO and GitHub.

    It's a dead simple extension but I use it constantly. Anyways, I redid Queue to use the web-ext-template and cleaned up a bunch of code stuff. I really need to stop making these big commits that just overwrite everything, but the JavaScript ecosystem sure doesn't make it easy.

    I also published an npm package to help with migrating data (like user settings) over from version to version, pretty much a super basic implementation of database migrations but for JS stuff. I initially did this in Queue a while back but I took it out, made it generic and capable of using any kind of versioning, and now hopefully it will be useful to me and maybe others for different projects.

    And I also got accepted into the GitHub Sponsors program a few weeks ago! 😊

    3 votes
  5. [8]
    Liru
    Link
    I've been working on a program that would effectively allow a massive reduction in needed hard drive space to archive lots of files. As a consequence of that, I've recently gone into a file format...

    I've been working on a program that would effectively allow a massive reduction in needed hard drive space to archive lots of files. As a consequence of that, I've recently gone into a file format rabbit hole, looking into using something current that would suit my needs or designing my own. I haven't found much that can be reused, though.

    Boy, format design is HARD. Every time I think I have something good or efficient, I think "What if X wasn't a hard requirement" or "What if Y is a valid use case", which makes me completely rethink what I've already thought of and slowly drives me madder. I'm trying to convince myself to settle on using a SQLite database as the container for a proof-of-concept program, but it makes me a bit sad considering that it doesn't have a lot of the properties that I initially wanted. I'm currently doing a small write-up of hard and soft requirements that will hopefully help me out a bit, since this has been mostly in my head. May post it to the Q&A thread if I'm stumped.

    Not to mention that it's hard to search for/think of file extensions that won't clash horribly with something else. I think I can just pick, but I really don't feel comfortable using any particular extension. If I get over it, I may use .lis, .las, or .mia, but I'm still thinking about it.

    Cache invalidation is relatively easy, IMO.

    2 votes
    1. [6]
      Wulfsta
      Link Parent
      Why not use a file system like ZFS or btrfs with zstd compression enabled for all files? ZFS can even do some fairly hefty data deduplication if I’m remembering correctly. Am I misunderstanding...

      Why not use a file system like ZFS or btrfs with zstd compression enabled for all files? ZFS can even do some fairly hefty data deduplication if I’m remembering correctly. Am I misunderstanding the purpose of your program?

      2 votes
      1. [5]
        Liru
        Link Parent
        I looked into it near the beginning. One of the hard requirements I have is that the result has to be easily distributable and usable for non-techies. Think something along the lines of competing...

        I looked into it near the beginning. One of the hard requirements I have is that the result has to be easily distributable and usable for non-techies. Think something along the lines of competing with .zip or .rar files, and not uploading things to a NAS for browsable storage.

        Even if I was to get ZFS working standalone and on files, ZFS' data deduplication works mostly on large identical data. My main use case is basically image files, and two files one would consider "identical" in terms of output pixels can have different representations on disk, so that wouldn't help much.

        zstd compression enabled for all files

        This is a frontrunner, but has some problems by itself which I'm trying to work around. A wrapper around .tar.zst seems likely for the final format despite some issues with it.

        1 vote
        1. [4]
          Wulfsta
          Link Parent
          Are you planning on doing anything like lossy compression on the images? For example, the data in a png that had been changed from a bit depth of 8 to 16 is the same, but that’s difficult to check...

          Are you planning on doing anything like lossy compression on the images? For example, the data in a png that had been changed from a bit depth of 8 to 16 is the same, but that’s difficult to check for - so identical output is a bit hard to deal with?

          1 vote
          1. [3]
            Liru
            Link Parent
            I'm trying to make it as lossless as possible, at the expense of initial conversion time. One problem I see is that a lot of the source images I'm going to be using are JPGs (high quality JPGs,...

            Are you planning on doing anything like lossy compression on the images?

            I'm trying to make it as lossless as possible, at the expense of initial conversion time. One problem I see is that a lot of the source images I'm going to be using are JPGs (high quality JPGs, but JPGs nonetheless), which will probably make me rethink a bunch of stuff. The only intentionally lossy things I may be doing is stripping metadata, and even then, I'm still thinking about if I should do that. (What happens after extracting the image data is going to be the user's choice, from lossless stuff to a "good enough" JPG.)

            For example, the data in a png that had been changed from a bit depth of 8 to 16 is the same, but that’s difficult to check for - so identical output is a bit hard to deal with?

            What do you mean by this? Unless I'm missing something, checking if an 8 bit colour depth was upscaled to 16 bits seems like something that would be easy to check for.

            1 vote
            1. [2]
              DataWraith
              Link Parent
              I'm not sure how well-known this is, but Dropbox's Lepton format can losslessly compress JPEGs by, as they claim, an average of 22%. JPEG XL also has a mode that converts old JPEGs into the new...

              One problem I see is that a lot of the source images I'm going to be using are JPGs (high quality JPGs, but JPGs nonetheless), which will probably make me rethink a bunch of stuff.

              I'm not sure how well-known this is, but Dropbox's Lepton format can losslessly compress JPEGs by, as they claim, an average of 22%.

              JPEG XL also has a mode that converts old JPEGs into the new format losslessly, with similar 20-30% savings in my experience.

              Unfortunately neither of those formats is well-supported currently, but it might still make sense to use them internally.

              3 votes
              1. Liru
                Link Parent
                I was aware of JPEG XL, but this is the first I'm hearing of Lepton. I'll try it out for comparison once I get some other things up and running, thanks a lot.

                I was aware of JPEG XL, but this is the first I'm hearing of Lepton. I'll try it out for comparison once I get some other things up and running, thanks a lot.

                1 vote
  6. Rudism
    Link
    I built a Corne keyboard over the weekend--my third one, but first wireless build using nice!nano controllers instead of Elite Cs, and first choc build using kailh low-profile switches and a flat...

    I built a Corne keyboard over the weekend--my third one, but first wireless build using nice!nano controllers instead of Elite Cs, and first choc build using kailh low-profile switches and a flat keycap profile.

    Been spending a bit of time in zmk, trying to get my keymap to match the one I use in qmk on the other two boards, which has been a challenge because I use both mouse keys and macros, which aren't yet officially supported in zmk. There's an existing PR that adds mouse keys, which is working fine for me, and I managed to find a work-around for my one missing macro by moving the functionality into a shell script that I invoke instead of having the keyboards play it out.

    Overall I'm happy with the new wireless board. The vendor I bought from offers a sleek aluminum case which was tempting, but my wife is already balking at how much I'm spending on this keyboard-building hobby so I decided to dust off the 3D printer and make my own case instead, which turned out nicer than I was expecting.

    I put a photo of all three boards here.

    1 vote
  7. babypuncher
    Link
    I found this amazing tool for applying an sRGB clamp to wide gamut monitors on Nvidia GPUs. It is really useful if you have a high end monitor that lacks a proper sRGB emulation mode despite good...

    I found this amazing tool for applying an sRGB clamp to wide gamut monitors on Nvidia GPUs. It is really useful if you have a high end monitor that lacks a proper sRGB emulation mode despite good factory calibration (like mine).

    I forked it with the goal of adding some new features, such as automatically turning the clamp off when in HDR mode, and re-applying the clamp whenever the display wakes.

    1 vote
  8. DMBuce
    Link
    I made a floating window manager for i3. Here it is in action: https://i.imgur.com/IVnZ9fS.mp4

    I made a floating window manager for i3. Here it is in action: https://i.imgur.com/IVnZ9fS.mp4

    1 vote