15
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
So I have been making good progress on my DIY MP3 player project this week. After struggling to find a good size e-Ink display with touch, I had to rethink my overall design and plan for the project. I have stumbled upon this YouTube video where Guy Dupont converts an old iPod into a raspberry pi based Spotify streaming device. The basis of the project involves using a Pi Zero running Raspberry PiOS Lite, Raspotify (headless Spotfify Connect program), and some custom front end programs to connect via Spotify API to control Spotify in a way that works with clickwheel navigation.
The iPod project felt like a good alternative to using e-Ink for my project. So, I started researching, and with a recommendation from a comment on last week's thread, I settled on a modern clickwheel alternative. I have not ordered it yet, as I am trying to bundle all the part ordering I need to minimize shipping costs. So, I am currently working on finding a small LCD display and a lithium battery. I have not used GPIO pins on a Pi before, so in sourcing these parts I have had to research how the GPIO pins work. So doing that research on how everything will get connected is my current holdup.
Since I have figured out the base of the software stack, I have done a very basic prototype so far. I have switched to running PiOS Lite, since I no longer needed the desktop environment. I also have gotten Raspotify running, which the install was pretty straightforward, and it was well documented to switch the audio output device to my USB dac/amp. Once that was working, I used a battery bank to try a headless proof of concept. Overall, I am quite pleased with the audio quality of this project. It helps that I am using my old USB dac/amp the discontinued NuForce uDAC-3, which drives headphones well. Once I get the remaining hardware components shipped, I will implement the rest of the software stack that Dupont used for his project. I will probably have to modify his code to work with my clickwheel and I may gradually expand on the features of his frontend application.
3 weeks ago I posted about implementing an audio declipper algorithm.
It's not done yet, but it works and animating the process looks pretty damn cool.
Single threaded version in Python + NumPy is marginally slower than the original Octave implementation released with the paper, but I added parallel processing, which helps a lot. Though on my 10 years old i7 CPU it's still about 3x slower than realtime, and that's for a mono single channel: declipping 1 minute of stereo audio takes about 6 minutes. There is some space for optimization, we'll see how lazy I am about it, but considering that the algorithm is better than most if not all commercial tools, I think it's good enough.
Now onwards to 1. some cleanup (in progress), 2. a GUI, 3. packaging and release.
Thats pretty incredible. I'd like to hear the original audio after the declipped version to see if there's much of a difference.
I basically only do python because its easy, I don't need speed, and I don't have the time or want to put in the effort to learn something else... but how much of the speed issue is just because of python?
No problem, the original audio is the beginning of this live recording. The original is cleaner, after direct comparison it's clear that some distortion is left in the declipped version, but I don't hear any artifacts otherwise.
Most of it. I use libraries written in C for the heaviest stuff of course (fast fourier transforms), but the overall algorithm is a python loop and afaik even just sending the data between the python interpreter and the C libraries or doing conversions between python internal types and numpy C types can slow the whole thing down by a ton. Based purely on reading stuff online and no experience on my own I think it could be sped up by an order of magnitude by going with a fast compiled language with simpler data types, fewer allocations etc.
Using C libraries for the numerical computation also doesn't allow me to do some optimizations: the algorithm uses short time fourier transform, meaning it divides the audio signal into small overlapping chunks (about 200 ms long) and applies the fourier transform on them individually instead of on the whole signal at once. In many lightly clipped real signals I could probably find some 200 ms chunks that do not contain any clipping and simply not process them at all. But since all of this is happening in a C library I'd have to fork it, write it in C and recompile it, which is not happening.
Since the algorithm isn't suitable for realtime (stream) processing anyway, I don't think it's a huge problem. But I might try rewriting it in Julia later, that could be a good compromise between speed and not having to learn C++ with its complexity or Rust with likely non-ideal math libraries.
This is super cool! Do you have a link to any of the papers you used as reference? I'd love to learn about how this works.
Sure, I implemented a subset of this one: https://arxiv.org/pdf/1905.00628
Here's a decent comparison of existing methods by the same author: https://arxiv.org/pdf/2007.07663
I still don't understand some specifics of the math involved, but the gist is that some time ago someone discovered that an unclipped audio signal has a high value of a criterion called sparsity and a clipped signal doesn't. Practically what this usually means is that you are seeking to gradually change the signal so that you increase/decrease a metric computed from the coefficients that you get by converting the signal into frequency domain by fourier transform, while applying some additional constraints on them (like "parts of the audio that are not clipped have to stay the same"). And there are various optimization algorithms to do that.
Different papers use different metrics, different constraints and different algorithms. This one is interesting because it adds some frequency bias - in real audio signals there's pretty much always more energy in the low frequencies than in the high frequencies, so it reconstructs the audio with this assumption, reducing audible distortion considerably more than an unbiased algorithm.
I've finally managed to get my stupid sprite moving routines figured out. It turns out that my studying unity to teach a class actually gave me some pointers as to how to organize my code in a way that helped it make sense. It feels like now that I've got that under control that everything else will be pretty easy to deal with, but that's probably just hubris.
Because this video game project was so far outside of my experience, I started to use Github Copilot. And boy, is it a mixed bag. The automatic code completion is amazing - it's basically writing the code that's in my mind for me, skipping over the difficulty of having to type it out - I've got a new keyboard and I'm going through an adaptation phase.
At some point I'm probably going to have to shut up about this because my intentions are to open-source the game under my real name for career reasons, and I'd prefer to not dox myself. I'm planning on making a proof of concept game and including those art resources as part of the package so it's actually useful for other people to build their own games on top of. Just let it be said that if someone with a new account posts something about a game similar to what I've talked about in the past, it's not me. I wouldn't be that dumb or full of hubris. I'm a responsible adult.
How much of Copilot's output do you typically have to adjust? Also what keyboard did you get if you don't mind me asking?
Hmm. It’s kind of hard to say. The suggestions it gives as I type are rather hit or miss, which is to be expected as it doesn’t actually know what I want. If I hit tab to accept it I will often have to go back manually to adjust small things like variable and file names. But using the chat functionality where you ask it to fix or explain code hasn’t been terribly useful. I haven’t used that too much but so but the problem is that it’s confidently incorrect about things. The other day I had an issue with a relatively convoluted iteration structure going through the first article of a table twice. Copilot gave me three wrong answers before I realized that the problem was just that I had written the instructions in the wrong order, but I got to that by leaning hard into the debugger; copilot was completely helpless. Though to be fair, I still didn’t entirely grasp why the code was behaving the way it was even with my error.
It also doesn’t seem to be that great at explaining a piece of code; the tone is rather unusual which makes it difficult to parse. This might be a bit unfair since I’m basing it on my own code, and that functionality might be intended to explain larger externally written code.
I got a Keychron K10 Pro I got on sale from Microcenter. I rather liked it at first but it’s very tall, so I desperately need a good wrist rest, and I am finding the key switches less enjoyable to use than I had thought when I was trying it out at the store. They’re a bit too sensitive and I am not as willing to deal with the lack of an actuation click than I thought I would be.
This is very informative, thank you!
This keyboard looks nice. I wonder what those Playstation-like keys are on the top right. Re: the switches, did you get the red switches? If you have the hot swappable version, you could give gateron browns a whirl, which I think has a low-profile version, which could pair nicely with low-profile keycaps... except that would be another expense.
Yeah, I got the reds. It is hot-swappable and I purchased a switch tester off of aliexpress to get a sample of different switches (keychron sells them on their website too, but they are out of stock). Though, honestly, if I couldn’t get the full experience of a switch in the store I don’t think getting them in this format is going to be that helpful either.
As it so happens I have a few leftover low profile switches left over from another project and it doesn’t look like they will work with this keyboard because the alignment posts on the bottom are keyed in a way that would not let them sit in properly. Either way, it’s really the base that is the problem here. It’s got a nice heavy weighted base that I love but it’s just too high to use without a wrist rest.
In any case I probably won’t be replacing the switches any time soon because a full set of them is likely to be about as much as the keyboard, and I wouldn’t describe my income as “bountiful” or even “stable” at the moment.
I made a webpage that my friend and I can use to start up or deactivate our Factorio server. The server is a DigitalOcean VPS that's paid for on an hourly basis, so I don't have to pay for it whenever we have it deactivated. Not only do we save money during idle hours, but it's particularly valuable since we've been playing less frequently lately. Before, we felt pressured to play more often to justify the server costs, or have me deal with manual server creation/deletion procedures often. Now we can axe the server casually whenever and even my friend can start up the server whenever he wants without depending on my availability.
I'm a web developer, so part of my goal with this project was to practice using the same kinds of tools I'd use for real user-facing web apps. Otherwise I probably could've gotten away with sharing a pair of shell scripts with my friend (though that would mean sharing my DigitalOcean and Cloudflare DNS API keys) or putting something together in a workflow builder tool like Retool. I ended up making a web app using Typescript running on Cloudflare Workers & Pages for this.
Details
You still have to pay for DigitalOcean VPS servers while they're shut down, so it's not enough just to shut down the server. The server must be actually deleted in order to not be billed for it. The save data needs to be stored somewhere outside of the server so it doesn't get deleted too. Also when I create a new server, it has a new IP address, so I need to update a DNS record to point at the new server each time.
The DigitalOcean VPS is started with the Docker Linux image, an attached persistent 8gb ($0.80/month) DigitalOcean block storage device where the Factorio save data (and ssh host key) is stored, and a startup script which sets up a swap file and starts a community-maintained Factorio docker image.
There's a worker script hosted on Cloudflare Workers which exposes RPC methods for reading the current server state or setting the desired state. The worker forwards its method calls to a single Cloudflare Durable Object where the current power state and desired power state are tracked, because durable objects have good transaction support and websocket support (which I'm not using for the webpage yet but want to). When the desired power state is changed, it runs a Cloudflare Workflow to set up the server or delete it (and terminates any previously started workflow). I use Workflows because each process has several steps that may fail and need to be retried, and I want everything I make to be durable against server failure at any point (along the principles of crash-only architecture), which Workflows handles well too.
Details about the server creation and deletion workflows
The delete server workflow:
The create server workflow:
Then the actual website is maybe only a hundred real lines of code using Next.js, a fullstack React framework, and hosted on Cloudflare Pages. If you're not yet signed in to a whitelisted account, the page shows a Google sign-in button handled by the next-auth library, otherwise it shows the server's current state ("On", "Off", "Turning on...", or "Turning off...") and a button to toggle it. This project has a service binding to the Cloudflare worker above, so I can call its methods directly from React server components and server actions. (I'm very happy with how easy Next.js makes it to communicate across the client/server boundary and how easy Cloudflare bindings make it to communicate across the server/worker boundary.) The turn-on/off button works through a normal html form, so the page doesn't even require the browser to use javascript except for auto-updating the displayed status.
Sounds interesting, what's your approximate monthly cost for such a thing? Also curious of the specs of the VPS that runs the server.
What's the spin up time?
I'm paying $5/month for Cloudflare Workers & Pages (though that's shared with some other small projects of mine). The VPS is a dual core 2gb ram $21/month instance. (Now that I'm not paying for that for the whole month, I might try upgrading that to a more expensive instance which might be able to handle game saving and players connecting faster.) So it's not huge savings, but if we decide to add anything else like a Minecraft server for us then this system will really shine.
The spin up time seems like 2-3 minutes usually.
Have you shopped around for VPS? I have a VPS for 5/month with 100gb storage, 5gb ram, 3 CPUs, 20TB bandwidth.
Seems like a lot of work and potential fail points just to save a few bucks each month. I know part of it is probably to gain the experience and know how to do such a thing... But when it's so little money saved vs time devoted, I start to question the return on investment except for the skills gained.
I used to have a problem of accumulating multiple game servers for things I rarely used, deciding to shut them down after 6 months just for the next week to be when friends and I get the sudden itch to play it again, and then on multiple occasions discovering they had failing hardware by the time I used them again, leading to support tickets and waiting days for a new server and trying to understand how to reconstruct everything on a new server, so to me it's particularly valuable and freeing to have things fully automated for setting up a new server or spinning things down. Especially since my costs won't go up as I add any number of games to the mix. I am a little interested though if there are any good cheaper providers that still have pay-by-the-hour servers that can be created or deleted through an API, though at the small fraction of the average day I'm actually using the game server it wouldn't move my costs much.
The first time I tried duckdb it was very slow (around 2020-2021). So I ignored any articles about it because it just seemed like marketing fluff. But recently I tried it on a few different things and I've been blown away.
So this week I'm taking a closer look at it and seeing if I can completely replace SQLite in my projects. If so, it should be pretty easy to write a function that upgrades any existing SQLite databases to duckdb. I don't think duckdb supports multiple-writer processes though so I'll probably still use SQLite for at least a few things
I am curious, why are you trying to replace sqlite?
Anyway, duckdb is pretty great for analytical roles. Back when the SSN leak happened, I wanted to get it into a database to get some statistics for it. Sqlite fell flat on its face, and Postgres was far too slow and used up way too much storage space (even on a very fast computer). Duckdb compressed the data a fair bit, something Postgres and sqlite don't do at all by default. And it is able to generate statistics based on that data set almost instantly. Very impressive bit of software.
Edit: I found this project to convert sqlite to duckdb. Might work for you out of the box, or as a template for your function.
Ah yeah duckdb supports SQLite out of the box so you can actually convert from SQLite directly to parquet with something like this:
For my use case I would read the first few bytes of a file and check if it is SQLite, if so, convert to duckdb or parquet. But the project you linked to is helpful. It looks like to go from SQLite to duckdb is around 3 lines:
The
install
andload
commands for built-in extensions are not actually needed (autoload_known_extensions
andautoinstall_known_extensions
are bothtrue
by default https://duckdb.org/docs/configuration/overview.html)I saw this article recently about a way to potentially do multiple writers for duckdb
https://www.definite.app/blog/duck-takes-flight
edit: to expand, this works because duckdb does sorta support multiple writer threads https://duckdb.org/docs/connect/concurrency
I've finally got around to releasing the source for the terrain generation library I built. I'll try to do the same with a gui I made to go with it soon: https://github.com/karsaroth/peakgen
I've been trying to learn about compression algorithms as a prelude to implementing some file formats. So far I've implemented run-length encoding and have an arithmetic coding implementation which I'm currently debugging
I've been interested in learning basic strategy for Blackjack, and found that all the existing apps for it seem to be essentially just flashcard apps. None that I found seem to actually implement games of blackjack and just tell you when you pick a non-optimal move. So I whipped up a little in-browser javascript game with my own stylised card designs. Puts a big annoying red X on screen if you go against strategy to make sure I remember every wrong move I make. Even has (in progress) settings to toggle doubling after splits and if the dealer hits on a soft 17, things that actually effect the basic strategy table in a few situations. I'm not likely to start actually gambling so I'm happy to report it's a fun little thing to pick up and put down.
I've been slowly getting more and more interested in radio stuff and SDRs, so after putting together a dual FlightAware/FlightRadar24 ADS-B collection station in December, I've been playing with making my own custom antennas. It's surprisingly simple to get something that performs reasonably well.
I don't have a very good view of the sky from where I live, so it's been really interesting playing with some cheap Amazon/Aliexpress antennas and the ones that I've built for myself. Currently have a nanoVNA-H4 on order from Aliexpress to start being able to look at the performance of what I'm building in real numbers rather than just "how many aircraft/messages can I receive?". If anyone else is interested, I'd be happy to write up a little more about what I've been doing and why it's been fun for me!
A very short "project," but for a class I'm in, we have to use Jupyter Notebook. On Windows, the process (as recommended by the instructor) was to open the Anaconda Navigator GUI, set the active environment, click on the button from there that starts up JupyterLab, and have it open in a web page in your browser. I wanted it to open in a PWA (so it would be more easily accessible when viewing other resources in my browser), and also skip the Anaconda UI if possible. So with a little bit of effort (and admittedly some help with ChatGPT), I put together a powershell script that does all this for me.
My (no-longer-quite-so) little Minecraft mod is coming along. It's taken a lot of code, potentially more than it should have, but one of my goals was making it fit in with the vanilla game and that means making sure all possible interactions work as intended.
It turns out the combinatorial explosion of possible states the block can be in still exists, but it's something the game figures out at launch rather than in my code. That screenshot is from a debug world that lists out every block in every possible block state and organises it in a square. I bumped my render distance up to the maximum of 32 chunks and it's still probably larger than two entire render distances in each direction. I've also got to a place where I'm happy with all of the models/textures, which you can also see in that screenshot. It took a while, but I think all of the variations fit in nicely with the game's style.
It's probably not too long before I release this mod, I'm starting to think about things like mod descriptions and screenshots and icons and showcases and all the rest of that stuff. It's been fun to work on, but I'm definitely ready to release the 1.0 version and let it exist for a while.
For some months, I've been idly replying about a project I've been slowly working on, a bash script powered by
yt-dlp
that mirrors content from YT to Plex.There are already other solutions for this, such as OpenTubeSync. But the issue I was experiencing is that they all mirror content as their own standalone file, so on the Plex end, content could only be added as a Movie library. This was less than helpful for curating/organizing content, as I wanted content to be grouped, not each item its own thing.
My script manages content with a TV library. Each channel is its own series. Videos are cataloged chronologically by year, with support up to 999 videos per year. So each video can be S2025E001, S2025E002, etc up to E999.
Organizing this way makes it much easier for me to see when something is added by a content producer, it'll show up on On Deck, and such.
I baked in a number of features, as well. Filtering in/out past live broadcasts, as well as shorts. Maximum resolution per source. SponsorBlock marking or removing. Requiring SponsorBlock availability, or optional availability with file upgrade if/when it becomes available. The ability to mark files from sources as watched on import (e.g. videos from your "Liked Videos", you've clearly already watched, so those can be marked as watched on import). Also re-creating playlists as either Plex Collections or Plex Playlists based on their public visibility.
I also created my own metadata management, eliminating any need for a third party Agent for the library. The script will locate creators and files within PMS, and manage their images, titles, descriptions, on its own.
I've been testing it for a few months now, it's mostly fine. I need to fix sorting of files within private playlists. And I'm about 50% done adding support for audio only files/a music library. But it's come together pretty well, for something I definitely should have done in a real language instead of bash.
I made this!
Not the game, and I got a lot of help on the port, but I took a version of the game and designed a process to get it to be played on something it was never intended for, and that's pretty cool.
How do I play this? I need to download a zip and apk and place them in the right spot? That's a lot of trouble just to try it out. Any easier path?
Edit: I'm on a phone, maybe it is easier on a desktop machine.
It’s a runtime for a portable game device that’s compatible with Portmaster, like an Ambernic or other arm based SBC. This one you would have to buy the game and side load it onto the device by putting the Android apk into a specific folder so it can unzip it, take the assets and run them in its own version of the Godot engine.
If you’re just using a phone, you could purchase the game from itch.io or the Google Play Store. There’s a Steam version as well for PC.