What programming/technical projects have you been working on?

I've relaunched my personal site: https://meerific.com

It looks like a personal blog, but there's quite a bit going on behind the scenes. All the static content lives on two folders on my computer: one Backblaze B2 folder (for images) and one Dropbox folder (for written content). The website turns these folders into a blog and image gallery. Any changes I make to the content is reflected on the website within a few seconds. Additionally, any images are processed via ThumbHash, and the resulting placeholder hashes are stored in an embedded database for faster initial page loads.

The site features full-text search as well as a storefront for photo prints. The storefront is turned off for now until I find a payment provider that has IPv6 endpoints.

As the site is written in Elixir, there's very little JavaScript -- aside from the websocket connection required for reactive features. If you're wondering why there's some latency, I'm running this on a cheap IPv6 Hetzner VPS (3.20EUR/month) based in Finland. I suppose I could use a service like Fly.io to get lower latency, but it's not super critical to do so. Besides, this is my site so I can experiment and do what I want.

The code for the site is here: https://codeberg.org/meerific/www

I got a bit further on the documentation for repeat-test, the property-testing library I’m writing. Here is Part 2. What’s slowing me down is that while writing the docs, I’ve been finding weaknesses that I want to fix.

I added a console.debugger() method that stops at a breakpoint in a failing test and console.sometimes(), which is an assert that has to be sometimes true and sometimes false in one of the test runs. It’s a way of checking coverage for random tests.

I've been tinkering with Stable Diffusion.

Also been collaborating on a Star Trek mod for Cosmoteer because every good sci-fi game deserves a good Star Trek mod. Also, quantum torpedoes go brrrrrrr (and I'm a giant Star Trek nerd🖖).

Over the past couple days I made a partial rust port of clustersort (one of the subcommands in xklb):

https://github.com/chapmanjacobd/clustersort

Rust is pretty easy to get started. The packaging and publishing experience is miles better than Python.

Unfortunately, my port is about 3x slower than the python version. The slowness comes from the KMeans clustering. sklearn uses BLAS but linfa said their code was performing "nearly as fast as" the BLAS implementation so they removed it to make the implementation simpler :/

Linfa has opt-in BLAS for pre-processing and it is about 8% faster... but that is only for the pre-processing part :/

It might be because I'm using a sparse array in the python version (spmatrix) but it looks like the rust library wants me to implement it on my own (sprs?).

After running cargo flamegraph it seems like the kmeans code is spending a lot of the time running ndarray::zip::Zip

It looks like there is a SIMD implementation of kmeans so that might be interesting to play around with as well

The most annoying parts of writing the code was that ndarray v0.16 seems to be incompatible with linfa 0.7 or something. I keep getting errors like:

mismatched types
`ArrayBase<OwnedRepr<f64>, Dim<...>>` and `ArrayBase<OwnedRepr<f64>, Dim<...>>` 
have similar names, but are actually distinct types
perhaps two different versions of crate `ndarray` are being used

Things would compile fine and then they wouldn't. cargo install --locked was working and then it suddenly had errors like above. I don't really get why it is being flaky. I didn't change Cargo.toml but I guess Cargo.lock decided to suddenly update?? Makes me appreciate duck typing.

But while investigating the source of slowness in rust I got some ideas for how I could change the Python version so back to Python for now~~ :)

I've been learning the Zig language during the past few weeks. Overall, it's a nice language that bills itself as "a better C", but I do occasionally miss the safety provided by the Rust borrow checker. I've made mistakes with pointers multiple times -- mistakes that the Rust compiler would have caught -- and gotten obscure error messages or segfaults as a consequence. Segfaults are kind of hard to debug for someone without the skill to wield a proper debugger, because all you get is a "Segfault at address xyz" message and then the program aborts.

The standard library is very cool and makes heavy use of the comptime feature. Zig allows you to execute emulated Zig code at compile time, and that can be used to operate on types. For example, the built-in std.ArrayList function takes a type parameter (e.g. u32) and then gives you a specialized ArrayList(u32) type that can store unsigned 32-bit integers, similar to Rust's Vec<u32>. It somehow feels more elegant than Rust's approach to generics, though I can't quite say why.

My learning project was to implement yet another Battlesnake, because I tend to do that for every new programming language I try. It's a nice way to exercise a language's features and ecosystem because you need JSON parsing, a webserver and game tree search logic.

Besides the occasional segfault, everything was very smooth with regards to the language, though I'm disappointed that the playing strength of the Zig snake is not quite where I was at with the Rust version. I've got a few scripts that will run matches between different versions of my snakes and incrementally calculate ratings with a new variant of Elo, which I find to be fairly accurate when compared to a (much slower) maximum-likelihood estimate of the ratings. It's endlessly frustrating to write code that you think should be superior to what you had before, and then it just under-performs even in the local league, though.

Right now, I'm implementing a way to tune the evaluation function using an Evolution Strategy (DiscoveredES). That has worked well with Rust, so I'm hoping that this will close the performance gap when I finish it in Zig.