11 votes

What programming/technical projects have you been working on?

This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?

20 comments

  1. deing
    Link
    My newest project is a refactored and unified Python port of what used to be a bunch of slightly different Crystal scripts: SeqArtFetch, a near-universal webcomic downloader. I read quite a bit of...

    My newest project is a refactored and unified Python port of what used to be a bunch of slightly different Crystal scripts: SeqArtFetch, a near-universal webcomic downloader.

    I read quite a bit of them and although I follow most using RSS, I really like having the archives available offline, especially with the bad and expensive mobile connections around here. The tool parses an episode's web page, using cssselect to filter for the "next page" link and art image(s), downloads the latter into a folder with a sensible naming scheme and continues on the next page until it hits a selector it recognizes as "last page". It has persistent configuration, so you can interrupt archive downloads and have them continue later, or just download the newest episode without extra commands.

    8 votes
  2. [4]
    tesseractcat
    Link
    I was inspired by the XKCD mirrorboard layout, which theorized a way to type with one hand, but without a large learning curve by flipping the right side of the QWERTY layout to the left. At the...

    I was inspired by the XKCD mirrorboard layout, which theorized a way to type with one hand, but without a large learning curve by flipping the right side of the QWERTY layout to the left. At the end of the article, they discuss a potential auto-prediction method with Regex to search through all possible words and see which one could be typed with that combination of flipped and unflipped keys, something like /[wo][ow][ru][dk][sl]/. I was pretty curious to see how this would work so I created a basic experimental version as a webpage, and also as a Chrome extension (I do the majority of my typing on ChromeOS, and I didn't want to delve into the ChromeOS input method APIs).

    Surprisingly, there are less collisions than I would have expected, and I can reach about 40% of my original typing speed with one hand. Some more common collisions I noticed were "words/works" and "like/side", but these are mitigated by tabbing through the possible options.

    7 votes
    1. [3]
      ThatFanficGuy
      Link Parent
      Uncaught TypeError: Cannot read property 'toLowerCase' of undefined at (index):70 at Object.handleEvent (<anonymous>:1:3001) The engine can't apply .toLowerCase() from word_to_match from...
      Uncaught TypeError: Cannot read property 'toLowerCase' of undefined
          at (index):70
          at Object.handleEvent (<anonymous>:1:3001)
      

      The engine can't apply .toLowerCase() from word_to_match from window.onload.

      By the way: use event.key: they're more reliable due to representing the key (as in, "Tab" or "Backspace") rather than keycode (which may be mapped differently on different systems and keyboards).

      2 votes
      1. [2]
        tesseractcat
        Link Parent
        Weird, that's an unreproducable error for me. There's no variable called word_to_match, although there is one called word_to_predict. I've only tested this page in Chrome, and it was simply a...

        Weird, that's an unreproducable error for me. There's no variable called word_to_match, although there is one called word_to_predict. I've only tested this page in Chrome, and it was simply a prototype to test out one handed typing, so that might be why it doesn't work. The event.key/event.keyCode distinction might be the reason why it isn't working, although I can't see why it would cause that error.

        2 votes
        1. ThatFanficGuy
          Link Parent
          Bad call from me. My short-term memory failed me. :) Have you tried entering a single letter and pressing Tab? I'm pretty sure it's because you haven't yet done whatever it is you do after user...

          There's no variable called word_to_match, although there is one called word_to_predict

          Bad call from me. My short-term memory failed me. :)

          Have you tried entering a single letter and pressing Tab? I'm pretty sure it's because you haven't yet done whatever it is you do after user presses Space. The function shouldn't run at all if there isn't a word to process yet. Try early return;.

          3 votes
  3. [3]
    Grendel
    Link
    For a while now I've been playing on and off with a project to try and generate country music lyrics using machine learning (this all started from a joke about how simple many modern hit country...

    For a while now I've been playing on and off with a project to try and generate country music lyrics using machine learning (this all started from a joke about how simple many modern hit country songs are).

    I picked it back up again recently using the GPT-2 model recently released by the Open AI research group. I scraped the lyrics for Billboards top 100 country songs for the last 20 years and fed that into the model for fine tuning.

    My poor little laptop had an OOM error when trying to train with these massive data sets, so I rented an AWS server to train it (cost me less than $10 by the end.). Nothing really great has come out of it, but it's been fun and has generated some pretty comical stuff.

    6 votes
    1. [2]
      antiolrach
      Link Parent
      Do you have any examples on hand? Seems like an interesting project, even if it didn't work perfectly

      Do you have any examples on hand? Seems like an interesting project, even if it didn't work perfectly

      3 votes
      1. Grendel
        Link Parent
        Here is a cherry picked example of one of the better examples :) I got a home down on the river I'm married to the woman I love Some days you gotta dance Live it up when you get the chance 'Cause...

        Here is a cherry picked example of one of the better examples :)

        I got a home down on the river
        I'm married to the woman I love
        Some days you gotta dance
        Live it up when you get the chance
        'Cause life is right and love is right
        I got a little bitty heart that's proud as a rock
        I've got a big blue flag waving down
        From a little bitty perspective
        Lookin' at me up close
        I think you'll see
        I'm the son of a gunnin'
        I got a right to bear arms
        'Cause life is right and love is right
        Standin' proud as a rock
        Standin' proud as a rock
        I'm proud as a rock
        I'm proud as a rock
        
        4 votes
  4. [5]
    ThatFanficGuy
    Link
    I've been creating my own keyboard layout. Using Ilya Birman's typography layout has been quite pleasant. It's been in my inventory for about a year now, and the only problems with using it came...

    I've been creating my own keyboard layout.

    Using Ilya Birman's typography layout has been quite pleasant. It's been in my inventory for about a year now, and the only problems with using it came from Windows itself.

    That said, I find it inadequate for my needs:

    • it lacks the umlauts (ä, ö, ü etc.), which is important for me if I'm ever gonna get better at German
    • the dash I use most often behind the ordinary - dash – the en-dash that I use for unbracketed parenthesis – is behind three-key stroke
    • some of the other symbols I'd rather use are not available at all, and things I see no way of ever using are surprisingly-prominent

    Basically, if it's difficult to get to, I'm never gonna get to it – and if it's easy to do, I'm gonna do it all the time. I'd much prefer my tools to accomodate my nature, not the other way around.

    So, I've been using the Birman layout as a base for my own.

    It's incomplete still. So far, I've compiled a few alpha builds using Birman's source code with changes and used them in practice. Right now, the layout – Spark – makes some changes to the Birman base – like putting umlauts to appropriate keys (so that [AltGr]+[a] produces ä).

    In the future, when I devote time to learning how to do so, I'd like the layout to become multi-layered. I'm thinking of replacing Caps Lock with some sort of an auxiliary key that would allow me to traverse the layers of the layout. Each layer would host its own category of keys/symbols.

    Low-level thinking permits adding media control keys, like zoom, or play/pause, or fast forward, or save screenshot. These might have to be bound to virtual keys that don't exist within the hardware of the keyboard.

    In terms of languages, I may well bind different keyboard layers to layouts of languages I aim to use regularly – like German, or Norwegian, or Icelandic and French that I aim to learn eventually.

    Perhaps with the assistance of software, I might be able to generate layers that aid my programming and design (e.g. by pasting the default index.html structure as plain text). Maybe add symbols that aren't used regularly but could aid production and/or presentation (such as the Unicode drawing boxes that I found useful in plain-text diagramming).

    The only part of software assistance I use right now is the Spark External module for AutoHotKey. I managed to compile it to an .EXE file that I load on boot. It replaces native key commands with the ones I set, which are mostly media controls for the audio (because I listen to music for 80% of my uptime if I can help it). I wanted it to do a little more, but I couldn't seem to figure out the language before putting the project on ice. (A downside to .EXE-module approach is that sometimes Windows seems to "offload" the module, causing it to stop working. A simple relaunch fixes it for a few days.)

    Ideally, I'd want to make my own keyboard, tied intricately to my needs. Media controls, multiple layers, hardware-level macros – who knows what else. I've seen various such projects online, and I'd always wanted to get into electronics.

    5 votes
    1. [2]
      deing
      Link Parent
      If you're looking for some inspiration, neo2 (site currently in maintenance mode, it has a lot more information usually) is a German-optimized layout with six layers, and hence a lot of otherwise...

      If you're looking for some inspiration, neo2 (site currently in maintenance mode, it has a lot more information usually) is a German-optimized layout with six layers, and hence a lot of otherwise inaccessible symbols and symbol placement choices that are at the very least much preferable to QWERTZ's. I'm using it to type this right now :)

      4 votes
    2. [2]
      hungariantoast
      (edited )
      Link Parent
      It seems like everything you want to accomplish would be more easily and powerfully done using something like the QMK keyboard firmware. Media keys, layers, macros for HTML boilerplate, that is...

      It seems like everything you want to accomplish would be more easily and powerfully done using something like the QMK keyboard firmware. Media keys, layers, macros for HTML boilerplate, that is all possible and relatively simple to set up. With a keyboard like the Planck, most of that stuff is even present in the default layout.

      If you want to build your own keyboard, then I recommend taking the easier, better route of buying a PCB board like the Planck, Preonic, or XD75. Alternatively, if you just want to buy a pre-built keyboard and not build it yourself, the ErgoDox EZ is great.

      Also, you might have noticed, all of those keyboard PCBs use hotswap sockets, so you literally plug your keyboard switches onto the PCB, no soldering required. It is much quicker than having to solder switches. You can literally pull and replace switches in seconds instead of having to desolder them.

      If you do not mind soldering your switches though, other PCBs like the BFO-9000 and Let's Split are great as well.

      Finally, you could just hand wire your own keyboard with a microcontroller like the Proton C.

      Personally, I would recommend just getting a hot swappable PCB and starting from there. It makes picking the right switch for your keyboard much less daunting and you can change them out any time. I own a Planck, so I am biased towards that keyboard and other OLKB stuff, since you can do a lot of wacky, crazy, wild, stuff with them.

      I would also like to note that I do not necessarily recommend the "Planck EZ", since it does not give you the option to pick between the MIT and grid layouts. Instead, it forces you to have a two unit spacebar with no option for a full, forty-eight key grid layout. I prefer the grid.

      That is a lot of info and I have no idea how much you know about this world of keyboards, so if you have any questions let me know and I will try to answer them.

      4 votes
      1. ThatFanficGuy
        Link Parent
        Bitchin'. Good info. Thanks a lot. I'm going to have to fuck with the QMK Configurator a bit to find the layout my laptop has (Acer Aspire ES1-311 series, if you wanna give me a hand). I've been...

        Bitchin'. Good info. Thanks a lot.

        I'm going to have to fuck with the QMK Configurator a bit to find the layout my laptop has (Acer Aspire ES1-311 series, if you wanna give me a hand). I've been browsing the configs for a while, and the fact that it's a 60% keyboard doesn't seem to hold any weight in the QMK world.

        if you have any questions let me know and I will try to answer them.

        I will bookmark this comment and return to it once I'm at the keyboard-building stage.

        2 votes
  5. DataWraith
    Link
    I've been using PyTorch to train neural networks to do OCR on and off over the last few months. This was prompted by a problem we had at work that was not solvable using the open source OCR engine...

    I've been using PyTorch to train neural networks to do OCR on and off over the last few months.
    This was prompted by a problem we had at work that was not solvable using the open source OCR engine Tesseract due to a noisy document background.
    So I thought I'd apply my nascent machine-learning expertise to the problem in my spare time.

    OCR of machine printed documents is apparently considered a solved problem, so there is surprisingly little information to be found that goes beyond "use Tesseract".
    There are a bajillion scientific papers about text spotting (finding text in natural images) and scene-text recognition (recognizing said text in images once cropped) though. Most of them are kind of complicated...

    The simplest approach I found, Facebook's Rosetta, simply wires a convolutional neural network (Resnet18) to directly output character-class probabilities for each vertical slice of the image using CTC -- Connectionist Temporal Classification. That's a method that optimizes a neural network to output the correct character sequence without one having to annotate where each character is in the image.

    However, since CTC decoding ignores doubled characters (the "ll" in "hello", for instance), the networks I ended up training had a lot of difficulty producing them. CTC introduces a blank character to separate such doubled characters, but the network almost never managed to produce it, for reasons I have yet to understand.

    I was stuck for a long while, until I reluctantly gave up on the idea of simplicity and brought out the big guns: encoder-decoder LSTMs enhanced with an attention mechanism.

    And that finally worked.

    The next step is to apply the network design to the actual problem, not to the easier proxy-task I've been using for development.

    4 votes
  6. skybrian
    Link
    The second prototype for my accordion bass electronics project is coming along. I'm building a 9-bass out of FR4 with two levels of keyboard switches, with a cut up plastic dowel for the keycaps...

    The second prototype for my accordion bass electronics project is coming along. I'm building a 9-bass out of FR4 with two levels of keyboard switches, with a cut up plastic dowel for the keycaps and nails to reach the lower level key switches.

    In the process I learned why people spend money on drill presses and built a rig to drill holes in the center of the dowels. I'm still not happy with the keycaps and I'm considering switching to wooden dowels or resin molding next time.

    Now I'm taking it apart again to do the wiring.

    3 votes
  7. alexandria
    Link
    I've been busy writing an implementation of ed(1) in Assembler. I've written most of the base, in that I can read a file into the line buffers and perform operations on that line buffer like...

    I've been busy writing an implementation of ed(1) in Assembler.

    I've written most of the base, in that I can read a file into the line buffers and perform operations on that line buffer like insertion and deletion. I've written a version of pike's regex implementation in assembler too, which will be nice when I get to parsing input and writing the command interface

    3 votes
  8. [3]
    Staross
    Link
    Doing some bioinformatics with Julia, the Bio packages are super good, it's a joy compared to the alternatives (R, python, ...).

    Doing some bioinformatics with Julia, the Bio packages are super good, it's a joy compared to the alternatives (R, python, ...).

    1 vote
    1. [2]
      ThatFanficGuy
      Link Parent
      What sorts of things do you do, exactly? Collecting data about biological processes? Where? How?

      What sorts of things do you do, exactly? Collecting data about biological processes? Where? How?

      1 vote
      1. Staross
        Link Parent
        Mostly analysis of genomic data, DNA and a bit or RNA-seq. What's really nice with the Bio.jl ecosystem is that you can open most file formats with a unified interface and get out reasonable...

        Mostly analysis of genomic data, DNA and a bit or RNA-seq. What's really nice with the Bio.jl ecosystem is that you can open most file formats with a unified interface and get out reasonable types. With other languages you often have to combine a ragtag bunch of tools and you often spend a lot of times converting stuff and calling external processes. Having good performances is very nice too. The packages are still a bit low-level, but that's not an issue for me since I like to develop my own tools rather than use existing ones.

        2 votes
  9. Lawrencium265
    Link
    I finally got around to installing Volvo vida software onto my laptop so that I could diagnose a few issues with a car I bought quite awhile ago. Fortunately the knockoff dice unit I bought from...

    I finally got around to installing Volvo vida software onto my laptop so that I could diagnose a few issues with a car I bought quite awhile ago. Fortunately the knockoff dice unit I bought from China is working just fine. Unfortunately I didn't find any electrical faults with the transmission solenoids and now I will attempt to remove and repair the transmission valve body next week.

    1 vote