13
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
It's not "real programming", but I've been working on polishing up a bash script that uses API calls and
docker execcommands to Sonarr and Plex to update metadata for items in my library that are imported under "TBA" and don't have metadata at the time of import. So that's helpful (to me, anyways).Bash scripts are real programming friend. Don't sell yourself short
For years (off and on, mind you) I have been trying to get off the ground to making a VST Plugin using C++ and the Steinberg SDK. I have learned some basics of the interfaces and features, but could never fully get a plugin to build and be used. I looked at a framework like JUCE but the experience wasn't great for me and for some reason parameter controls felts wonky with their UI library.
I came across a Rust UI library on lobste.rs the other day (which I can't for the life of me remember what its called) and in the marketing material they mentioned a plugin company was using the lib for their EQ plugin. Something clicked in me and I started looking up VST wrapper for Rust and low and behold found
nih-plug.One night later and I have a really shitty sounding soft clipping plugin I can actually use in an actual DAW. I may be a Rust convert now.
The UI library sounds like it might be "Slint" 🤔
Huzzah, that's the one!
I have been having a bit of fun booting up a personal web based project using more modern frameworks, including Next.js and React. Previously, this dinosaur would generally go for setup with something like express and jQuery. I have been extremely surprised by how much easier React Components make it to create UI things. Specifically, recursive Components have been a delight to discover.
Related to that same project, I have been experimenting with the openAI API to explore an idea I had of LLM assisted organization of notes I make. Basically, with my ADHD brain I do need a note keeping/thought organizing system of sorts. I have a system that works but I can't resist the urge to try and automate some aspect of it in order to improve on it.
My weak points with any system I have applied is that it mostly serves as an external brain for whatever is visible to me. Which is why currently I renew a list of items on a daily basis. Organizing notes/thoughts consistently is something that I never really managed to do so.
So what I am currently experimenting with, is a note-taking system where I only have to input the note body. Then through some clever (I think so anyway) system prompts towards GPT potential titles and tags are defined as well as two summaries. One summary for me as a human and one summary that works good for LLMs. The LLM aimed summaries are then also used to effectively let GPT figure out if I made notes previously that are potentially related. All of this only happens note creation where I can confirm things like related notes, pick a title and potentially add metadata myself. The results will then be stored in a database and for stuff like note browsing and all that it is just a personal knowledge base.
So far the results have been quite promising. And because I am not making an AI tool where everything tries to be clever the API usage (in as far as costs go) also is kept in check and reasonably priced.
If I manage to finish this project I might explore using open source models, but I am not nearly at that point yet.
I've been working off and on this week on a CLI tool that opens a browser and saves arbitrary table-like data (including any JSON responses) that it sees into a SQLITE database so that I can easily filter through jobs based on my own where clause across multiple job sites.
I can convert arbitrary JSON to 3NF relational tables just fine. The problem right now is figuring out what the different tables should be named :/
Do you mean that you're having trouble naming your tables, or that you want the tool to automatically pick table names?
Yes, the tool needs to automatically make tables. There are two different conflicting ideals going on in my head:
If possible, try to get a construct a name from one or more of: (the endpoint that the response came from, the original object key that the array was a value for, etc). The benefit of this is mostly readability but also query comprehension.
Choose generic names so the longest, widest table will become t1, second table t2, etc. The benefit of this is that you won't need to look at the database before running a query. But also more queries might be transferable across different sites. I'm trying to remember how ogr2ogr does it because it does a similar thing.
I might need to do a bit of both, or hide option 1 behind a flag.
Huh, that's interesting. Are you saving JSON blobs in your db or how do you handle the arbitrary shapes?
I do something similar to this: https://github.com/tulip/relationalize/blob/main/relationalize/relationalize.py but I use auto-incremented ids instead of UUIDs
Just as an FYI, unless you really want a file based database PostgreSQL might make dealing with JSON a lot easier. As you can just have JSON as a data type and query based on the JSON in it. https://www.postgresql.org/docs/9.3/functions-json.html
Yeah I really like PostgreSQL, PostGIS. But it makes my program more difficult to install for other people. Python comes with SQLITE support out of the box. I wish there was a static build of pg built into python or something, even if that was limited to something like module-level database namespaces.
Right now I can easily install the same python package on my phone and everything* works great but switching to postgres would make it impossible to have that same level of portability.
* I haven't tested all the package modules on my phone but the point is the SQLITE ones work out of the box just fine. maybe the browser ones would require downloading
geckodriverto the PATH or more tinkeringAh yeah fair enough, for distribution to other people sqlite does make things a lot more straightforward.
Does your tool work on the standard job sites? If so, is your tool open source?
Ideally it should work on ANY site. HTML tables, table-like HTML data but not actually "HTML tables", JSON responses, protobuf, websocket...
Right now, I'm just focusing on JSON responses because it seems to be the most common.
It's 80% done but just fine-tuning some stuff before I push it up. In the next couple weeks look for the new subcommands
siteadd,siteupdate, andsitesql(basicallysiteaddbut as a temporary SQLITE :memory: database with immediate SQL query-ability) here:https://github.com/chapmanjacobd/library
I've been working on setting up https://coalesce.audio to autoscale. The collaborative editing server benefits from load balancing with session affinity, so all editors of a document are connected to (roughly) the same process. Envoy supports websockets with consistent hash routing by http header, so my plan is to use Kubernetes to automate scaling up and down the ring, with Envoy picking up the server instances via DNS. There's a bit of complexity to handling ring size changes, because there may be more than one process with connections while they drain. I plan to have the hosts broadcast via redis Pubsub to stay in sync, which should work pretty seamlessly because the collaborative editing is CRDT based. Hopefully this works decently, because I've been trying to avoid writing a custom websocket session manager. 😅
After months of trying to figure out the optimal SaaS techstack, I've finally given up and choosing to go back to plain old ruby.
It might be boring/inefficient/messy, but the productivity I feel is miles ahead of other frameworks including my main language js.
Soooo I'm binging gorails videos and hopefully learn enough to get something going soon.
If you count Steam API hacking, then I recently showcased some data science investigations into Roguelike data for an invited talk
The Steam API isn't too shabby for putting together datasets. Issues arise more with tagging systems based on (IMO) data quality. I'm trying to think of a product tagging system that makes anyone happy, but one underrated aspect of Steam's is that we can parse out the value counts of tags to leverage with cosine similarity scores for more accurate product recommendations.
I've got a grandiose plan to write a self-hosted system, written entirely in dart, capable of running generative AI models with various backends. My long-term vision is to be like a self-hosted ChatGPT+Automattic1111 interface that offers a convenient mobile, desktop, and CLI interface with curated workflows for various tasks, while allowing you to still peek under the hood and manipulate things at the lowest level.
For LLMs, something like having a chat interface you can toggle off and edit the context directly, swap sampling methods interactively, generate many iterations of a completion, and so on.
In the short term, I just want a convenient mobile app I can use to access LLMs running on my Mac Studio, with quality-of-life features like history and seamless human-in-the-loop completions editing.
Notable about this week is that I've managed to work out all the integration kinks with llama.cpp as well as dart's FFI and experimental native assets feature. I wasn't sure if this would even work in the dart server-side ecosystem TBH, but now I'm at a point where I can begin working on the backend server and the client apps!
So while I don't have anything to demo, I'm very excited about the next steps.
https://github.com/crasm/ensemble
I’ve always had a deep respect for LISP and other LISP-y languages because of their simplicity and power. The syntax is super easy to parse which is fun for me because I’m not any good at writing parsers.
Which is what I’m doing: I’ve been having fun writing my own LISP language. I just want to see how far I can take it before I either stop having fun or I start encountering serious friction.
As an aside, I had a more experienced coworker encourage me on trying to work on my brand which I’ve dabbled with for the last several years but never really took super seriously. At some point I’d like to write one or two articles about this project.