What programming/technical projects have you been working on?

I spent all last weekend in an ADHD dopamine fueled coding session working on my website redesign. For background, it is Audiobookcovers.com. One of the best subreddits is /r/audiobookcovers. The people there find or create artwork and assemble it into audiobook cover art. There is some pretty fantastic artwork there. Well reddit search is awful, and a few years ago I wanted a way to fix it. So I have been working on this website as a hobby project since then.

The original version of the site was some terrible python scripts hosted on pythonanywhere.com. It used a google cloud API to OCR the images and had fulltext search through either mysql or elasticsearch (can't remember which). It was very clunky, but I learned quite a lot. I have since rewritten it many times with different technologies. The currently hosted version is using Nextjs on Vercel, and is what I used to learn react. It uses openAI's CLIP model for the search, in a supabase postgres db.

If you haven't heard of it, CLIP is a really cool AI model. Ultimately it just generates embeddings. An embedding is just a fancy name for a vector, which is a fancy name for a long string of numbers. For example (111, 386, 99) is an embedding. Specifically, it is a 3 dimensional embedding. A CLIP AI model is a collection of two separate models, one for images and one for text. If you gave it a picture of a dog, and the text "a dog", the embeddings you get from the two models will be similar. That property is called "mapping to the same vector space". This property is incredibly useful for image search. You first get a list of images, run the image model on each, and store that list of embeddings. Then, when a user wants to search, you take their search term and run it through the text model. Then you take the search embedding and calculate the distance from all of the image embeddings, and show the closest ones. For those of you wondering, this isn't directly related to the GPT LLMs that are in the news nowadays. The CLIP model was first released before chatgpt existed. And, while it does take some computing power, it doesn't take anywhere near the power needed for an LLM, even a very small one.

This elegantly fixes a ton of problems with traditional image search. My original website used OCR, so you could only search for the text on the image. And if the OCR messed something up, it would be very difficult to search for. And if the image has no text, it just wasn't searchable. With this AI search, finding text from images pretty much just works. But you can also describe images, and that works too. Searching for "blue dragon" will give you images of the Eragon cover. It's a very cool technology.

Anyway, I tested multiple setups for this. The model needed a decent amount of ram (for a cheap server, 2-3gb). So keeping it running constantly on a basic VM host would be costly. I also tried a bunch of different on demand services, that will start your docker image when a request comes in. The big issue was cold starts. One of the best services was google cloud run, but starting the machine took 30+ seconds, and keeping a machine alive all the time would cost more than I wanted to spend. So I ended up on Replicate. It has the same cold start problem, but their public models are shared between customers. So if you use a commonly used model, you should never have cold starts. This worked fantastically for the current iteration of the website.

The issue with replicate is costs. Shortly after I coded this, I went back to school, and stopped working completely. The bills started being higher than I budgeted, so I set a $4/month spend cap. Unfortunately that meant the website would just stop working at some point each month once the spend cap hit. I was hoping to pay for this cost with basic google adsense, but I made about $1.50 in the year I had ads on the site, so that definitely didn't pan out.

That is all to say I have been working on a redesigned system that wouldn't use Replicate for quite a while now. Previously I was focused on changing the stack (Deno + Fresh), so I only tried to solve the replicate problem recently. Somehow I stumbled on Fly.io. I knew of them previously, but I didn't think they were a reasonable solution to my problem. If you aren't aware, they are a virtual machine hosting provider that is quite good at low-traffic websites, but can scale up very easily. But they don't just give you a VM and let you set it up from scratch, like ec2 or digitalocean. You give them a docker image, and they run that as a full virtual machine. I previously thought this seemed backwards. The benefit of docker is that they are fast to start, and don't have the overhead of a full virtual machine. Why would I want a full VM instead? Well they aren't using a traditional hypervisor, they use a technology called firecracker. It's the same tech that powers AWS lambda. It is built on Rust, and allows you to run very lightweight virtual machines with almost no overhead, and far better isolation between machines than docker provides. They somehow make cold starts on these machines incredibly quick; they are quick enough to be almost not noticeable even for a webserver. They also can suspend instead of stop them, which I think is similar to hibernate. You pay for compute and memory when the machine is running, but when it's off, you only pay a tiny amount for the root filesystem storage space. Very cool technology.

Also, as part of this rewrite, I wanted to look into newer CLIP models. There are a few, but the ones I am interested in are Apple's MobileClip. Apple released a series of CLIP models that are free to use. Unlike other models, they are designed to run on mobile phones, so they are faster to run than other CLIP models. I assume these are what power the iPhone's Photos search interface. They have 5 versions, s0, s1, s2, b, and blt. s0 is the fastest, and blt is the highest quality but slowest. In order to compare, I needed to regenerate the embeddings for my entire list of 5000 images with the new models. My original embed with openAI clip running on replicate cost about $2.50. It's a one-time cost, so I didn't think much of it. With the mobileclip models running on replicate, the entire embedding pass for all 5 models cost around $0.75. Needless to say, I am sticking with fly.io and not using replicate again. I think replicate is probably a good service for larger LLM or diffusion models, but for a small model like CLIP, its just the wrong tool for the job.

I am still in the process of comparing the models, but I really like the results so far. I found one search where the openAI model brought up very weird results. I don't know if the embeddings in my database got messed up or something. All of the mobileclip models perform better than the openAI model, and run much faster and more cheaply. I think that s0 is going to be suitable for my needs, but I am still planning to test more.

Anyway, this rewrite is also a rewrite of the website as well. I am using Deno as a runtime. Throughout my learning javascript, react, and web development, I know one thing for sure: typescript is essential. Typescript is my favorite language by a huge margin, and plain javascript is one of my least favorites. Learning TS has actually made me dislike Python because of it's lackluster type system. Before this rewrite, I made a Factorio mod, and have been working on a Playdate game, both of which use Lua. I think I like Lua, but I miss typescript's type system all the time. ts-lua would be such a fantastic project. So I want a runtime that has good support for typescript. I have never been impressed with ts-node or tsc, because setting them up was a lot of work. I really like Bun, but I have settled on Deno. With Deno 2.0 node compatibility, I don't see any reason to use Bun or other alternate runtimes. And for the website, I have been using the Fresh 2.0 alpha. It is a much different way of building a website than Next or client-side react, but, for the right project, I think it works well. With the rewrite, the new site is almost exclusively server rendered. I use a small bit of client-side preact to swap the images when they are loaded with the blurred placeholder images. If you have more than a tiny slice of client interactivity, Fresh is probably a bad framework, but for Audiobookcovers.com, it seems to work well.

Also as part of this, I wanted to stop using Supabase. I currently am using it as a plain postgres database. I like it as a way to host a cheap database, but I wanted to try hosting my own database. I have an Oracle Cloud free tier VM that has a 4 core ampere CPU, 24GB ram, and 200GB storage. This seemed like a perfect candidate for a database server which could be used for all my future projects. So I had to learn how to setup postgres, pgbouncer, and pgbackrest myself. I had done this once before, but ran into some issues. This time, I had help from Gemini pretty extensively. It hallucinated quite a bit, but I was able to work through the hallucinations and get it working. I also think I have it secured quite well. You have to connect through pgbouncer, have to use ssl, and have to use a scram-sha-256 password. I hope that is everything I have to worry about. SSH is only possible through tailscale or an ssh key that lives in 1password. Once you know all the pieces that go into it, it isn't terribly difficult to run postgres.

I also needed a new host for the images themselves. I currently use backblaze with fastly cdn in front. For the rewrite, I am trying to avoid services that increase price significantly to get out of the free tier. The fastly free tier includes $50/month of cdn traffic, but if you exceed that, you have to pay at least $50/month. Same issue with supabase, where if I exceed their limits, I have to pay at least $20/month. I am happy to pay, but I don't want the $0 to $20+ jump if my website gets a bit more traffic than usual. In my testing with fly.io, I found tigris data. It's an s3 clone built on top of fly.io. Storage costs about 3x more than backblaze b2, but has a global CDN and unlimited free egress. Migrating to them was very easy, and I didn't have to set up a CDN in front, which was nice.

My current iteration uses Drizzle for my database. It's a lightweight typescript wrapper for sql. You write functions that look very similar to sql in typescript, and have the benefit of full type safety. For example, a simple db lookup looks like this:

const [result] = await db
        .select({
          id: image.id,
          extension: image.extension,
          blurhash: image.blurhash,
          source: image.source,
        })
        .from(image)
        .where(eq(image.id, input))
        .limit(1);

Very slick, which is why I used it in the first place. But for more complex queries, it can get annoying to use quickly. For the embedding search, I need to order the results by a calculated column. The only way to do that in Drizzle is with a subquery. Here is what that query looked like:

const sq = db.$with("sq").as(
        db
          .select({
            id: image.id,
            extension: image.extension,
            blurhash: image.blurhash,
            source: image.source,
            similarity: cosineDistance(image.embedding, query.embedding).as(
              "similarity",
            ),
          })
          .from(image)
          .where(eq(image.searchable, true)),
      );

      const dbResult = await db
        .with(sq)
        .select()
        .from(sq)
        .where(lte(sq.similarity, similarity))
        .orderBy(sq.similarity);

Not very slick. The way I saw it, I wanted to just use basic sql, because I knew how to do that query in sql. But I needed to have type safety (see above). I considered building my own wrapper around a postgres node api that would validate the results with zod. But I did some searching, and someone already build exactly that: slonik. You give it a template string sql query, and a zod validator for what the result should look like. Exactly what I wanted! That exact same query looks like this in slonik:

const results = await pool.many(
    sql.typeAlias("imageDataWithDistance")`
      SELECT
        id,
        source,
        extension,
        blurhash,
        ${model.dbColumn} <=> ${JSON.stringify(vector.embedding)} as distance
      FROM image
      WHERE searchable
      ORDER BY distance ASC
      LIMIT 24
    `,
  );

// The zod validator is defined in another file:
imageDataWithDistance: z.object({
      id: z.string().uuid(),
      source: z.string(),
      extension: z.string(),
      blurhash: z.string(),
      distance: z.number(),
    }),

So much nicer! (Some of you who aren't familiar with modern typescript may be worried about sql injection, but that isn't an issue here. JS template functions use foo``, and send that template to the function foo. It gets the template string and the functions within the template, and assembles it itself. So a smart sql api can use that to convert it into a properly parameterized sql query.) Deno has a compatibility issue with node-postgres, which slonik uses internally. That took a lot of debugging to file a bug report, but I finally found a workaround for slonik that someone else posted on github.

This rewrite has been a ton of fun! I am looking forward to getting my image downloader and the frontend website finished, but I am hopeful it will be soon!