17
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I've been looking into new ways to manage a few different servers that I have. Over the past year, I've really enjoyed working with immutable updates provided by ostree in Fedora IoT. It's the same technology that is used in Fedora Silverblue (and the other atomic desktops).
One interesting technology that has come out from this is ostree native containers--which are essentially bootable containers. Really cool concept but in practice it is not all that needed because everyone is already used to using the Cloud--even something like Kubernetes is seemingly a lower maintenance burden than having to know some hardware details, right?* But I still think it is interesting so I've been digging into this area... I guess one of the main reasons why it is interesting to me is cost reduction. Even something like DigitalOcean's App Platform is 2x the cost of a DigitalOcean VPS and they only give you 100 GB of egress--the pricing is much more competitive if you manage your app containers on your own.
A couple years ago, in an effort to make ostree native containers more universal (to eventually support debian based distributions) the project bootc was created. Right now it is still a bit experimental and only supports containers which use rpm-ostree. It seems that bootc and dnf5 will eventually replace ostree native containers.
The workflow for using bootc looks like this:
For example:
podman build(orpodman -c podman-machine-default-root buildif you want to use podman-bootc locally)podman-bootc runif you want to test-run the created image vialibvirtbootc switchto boot into the built OS or upload the gzip'd qcow2 disk image to something like DigitalOcean as a custom OS imageFor example, to enable rpm-ostree auto-updates (if you aren't planning on updating the container registry which is kind of the point of deploying via container but maybe you don't know how to use PXE like me so you are using bootc instead of just using the default, more popular, OS images or learning k8s, etc):
This last point makes it a bit obvious why bootable containers are not super useful--but switching to frequent updates is flexible enough and the tooling might be more comfortable than something like virt-manager, QEMU, Virtualbox, etc. More details here. But if you are already used to using those technologies there is really nothing novel provided here.
During container-driven updates (bootc), your
/var/homeand/etcis unchanged (3-way merge when/homeor/etcfiles are changed within the container image); this is similar to existingrpm-ostreeupdates. More details hereIt's not possible to embed containers inside of bootable containers due to a limitation in the OCI container tar format--but you can write services that will pull and start containers once the bootable container has booted.
Here are a few interesting links I've been able to gather:
* I don't think k8s is all that bad but it's definitely trying to solve too many problems at once. Something lighter-weight like Nomad (to handle bin-packing) and bootc might work really well. But you might not even need bootc and you might not even need Nomad... etc
edit: I put together a more complete example of how to use it here:
https://github.com/chapmanjacobd/computer/tree/main/.github/bootc/server
Unfortunately, it seems like
bootc-image-builderwants the container image in/var/lib/containers/storagebutpodman-bootc runwants it inside of thepodman-machine-default-rootVM so you have to build it twice ?? :/I wrote an extremely barebones 3D printer slicer for implicit surfaces over the course of a weekend, so here it is, I guess! Much of the reasoning can be found in the README, happy to answer any questions.
WeekendSlicer
I looked through the implicit function website that is linked in the readme, but I’ll admit that it went way over my head. If you don’t mind, could you explain what makes this unique from typical cad and slicing? Like what can you do with this technique that you can’t do with current tools?
Anyway, this looks very cool! I couldn’t imagine building a slicer myself.
Well, the core difference is that this format represents volumes rather than boundary surfaces. That is, formats like STL and STEP only hold data about the boundaries of the objects they represent, whereas in this representation you actually have to figure out where these boundaries are (where the equation that represents the surface is satisfied). The arguably best technique used to do this (at least to my knowledge about the state of the art) is dual contouring. Since Fidget is extremely fast at this, it exposes simple manipulations of the representation (as long as the SDF is “correct”) to achieve things like determining the path of a perimeter. In my opinion, these should also allow for extremely simple manipulations for top and bottom surface detection, infill generation (of certain kinds of infill), and interesting opportunities for things like support generation. Broadly, it is interesting to me because I think these manipulations, possibly combined with some post-processing of the generated paths, would be easier to write and maintain than the alternatives that already exist. It is also not necessarily less computationally intense than current slicers, and writing transformations to go from boundary surface representations to correct SDFs is nontrivial, but I think doable. I will likely return to working on it if Fidget implements 2D path generation, so I don’t have to extract paths from generated 3D meshes, with the next milestone to be slicing a benchy.
I released v0.5.0 of https://github.com/robinovitch61/kl, my Kubernetes log viewer for your terminal.
I'm excited about optimizing it further and adding things like in-memory analysis of logs: grouping by similarity for anomaly detection, visualizations for log volume by time, etc
So around the start of last week I thought I would do something that is both a good professional show and an interesting project for me personally. I decided to make a video game from scratch.
Well, actually none of that is accurate. It’s not a good professional development because I’m writing it in Lua with LÖVE and I have zero intention of working for a big video game company whatsoever. I don’t think I’d even be trying to work for indie studios either because the design patterns I’ve learned in enterprise-land do not really work well for this!
Really, this is a very challenging project for me. This is going to be the first program I’ve written at this scale. Right now I’m having a hard time even figuring out where to start. I’m copying the mechanical game design from Shining Force so I’m not blowing in the wind creatively, but I’m finding it hard to figure out how to structure things because all of the tutorials I’ve found focus on how to do action games, which are surprisingly more simple (which is good in a way, because I want to experiment with a tactics/action game at some point in the future, but it really doesn’t help me now).
I’m taking something of a break from it at the moment because I need to make a few assets and make some decisions on how the game will look in terms of resolution and relative sizing, and before I could even get to that point I needed to clean up the room enough so that I could get to the monitor we have with the digitizer because I can only barely draw with a pen trying to use a mouse is madness. But the act of cleaning and seeing the end result has been good for my mental state because it feels like an accomplishment; a great thing to see after you’ve realized the scope of the work you’re trying to take on.
Somewhat ironically I will likely need to learn Unity for my job so I can teach it to children. That’s going to be fun and not make me regret deciding to make the game from scratch at all.
Hey me too! I took Thursday and Friday off this week and finally decided to get started.
I’ve wanted to try game dev for a long time but similarly, I have been so overwhelmed by, well, it. There are so many tutorials (and half of them are just over-monetized “like and subscribe” bait), should I try 2d or 3d, which of the ten half-baked game concepts in my head do I want to take a crack at, etc.. Basically the only thing I was sure of was that I wanted to work with Godot. I’ve wanted to work on this all year but I’ve had such bad analysis paralysis that I never got much past installing Godot.
I basically spent my whole day Thursday watching hours of entry-level tutorials on Godot, both 2d and 3d with various types of games. I don’t know that I retained very much of it, but it did get me more fired up to try some stuff. I started to do the same on Friday and the first tutorial I watched covered modeling, rigging, animating a 3rd person player character in Blender and then adding it to Godot. It didn’t seem as scary as I figured, so halfway through I started it over and followed along. By the end of the day I had a low-poly 3d dude that could run around on a big square plane with an idle animation that blended into a run animation on move. It’s nothing compared to the scale of even a small video game, but it was a big achievement for me.
I decided to push myself and try to apply what I had learned, so I created a jump animation and code and figured out how to override the run animation with the jump one when appropriate. Felt like a massive win for me.
I can’t draw to save my life, but it turns out 3d modeling makes some sense to me, so I guess that’s one question answered! I’m sure I’ll be singing a different tune as scale and complexity increases, but that’s a tomorrow problem 😂
Cheers to us taking the plunge! Hope you’re having fun with it and here’s hoping we can both stick with it and make something great!
Honestly, the only reason why I didn’t start with Godot was that I didn’t want to learn GDScript. But that would have probably helped with a lot of the framework bits that I’m trying to work on right now, so maybe I should have spent a bit more time evaluating it.
Yep I hear you, adding a language to the laundry list of stuff to learn is no small task.
I don’t do much more than basic scripting in my daily life so I’m going to be learning the language no matter what engine I use. So far GDScript just feels like Python and hasn’t really gotten in my way much yet. YMMV though!
I've recently been experimenting with Godot as well, and I've been enjoying it. If you're not a big Python-syntax/GDScript person, there are also .NET builds that let you write everything in C#.
The API is largely identical, just swap out the snake casing and a few other minor things. There are a couple of limitations (No export to the web right now and maybe not for a good long while), but for the most part it works really well.
I've been trying to work up the motivation to build a small game for fun myself. Years and years ago I made probably a few dozen little games ranging from ones playable in terminal to raw C++ (some with opengl, some with directx, some with SDL, etc.) to ones in Unity and I guess also a few in HTML5. But that mostly died off after getting a job. I did spend around 5 years working on games professionally though.
While I'm probably a bit rusty in game-specific stuff if you, or anyone else interested in game dev, has any specific questions feel free to DM me. While tutorials can do a good job with how to get started I think it's a lot harder to find answers to how things get made scalable. My game-related expertise is in gameplay programming, but I've also done a fair bit of custom engine stuff too.
As an aside, the thing that kills motivation for me most personally is that I have really no artistic skill to speak of. So if anyone has any ideas for mitigating that feel free to reach out on that front too. One idea I've thought would be fun due to the constraints it creates is to start with just the raw assets from a game I haven't played and at the end compare the direction I went with what the assets were meant for. I don't want to be stealing anything though so I'd need to find someone cool with that first.
I suppose the thing I need right now is less specific advice and more of a resource to learn more about architecture for video games. I remember a while back reading about how using ECS is supposed to be really helpful, but I don’t even have a full grasp on what that is or why it’s supposed to be helpful. So I suppose what I would like the most is a book recommendation.
Art is also a pain point for me. To the point where most of the people I’ve told I am making a game are have just told them I am making a game engine so that they don’t expect a full game out of me. I have commandeered my husbands monitor with a pen digitizer and hopefully not having to draw with a mouse will help release some of those creative juices.
I’m sure you’re both aware, but there are tons of CC0-licensed free assets on itch.io. If artistic direction or learning the process of creating the assets aren’t in scope for you at the moment, snagging an asset pack might help to reduce that friction, or at least defer it long enough that you can keep working and not get discouraged. I’d imagine it would also be a good way to “shop” for an artist you might want to commission bespoke assets from down the road.
Unhelpfully I'm not much of a reader so I don't have useful recommendations. On that specific topic though I have a surprising amount of information. In addition to me implementing a few different ECS systems my previous company invited a couple of people from Unity to give a talk and ECS was the topic of the talk.
Broadly there are 3 common ways to architect a game engine, although they are not strict boundaries and may overlap or use multiple within different parts of the same engine. While not the best names I'll refer to them Inherit, Compose, and ECS.
Inherit
A pretty standard starter design for anyone with OOP experience. A naive example of this could look like
Compose
Rather than inheriting from GameObject game objects are just buckets for data and behaviors, often combined together. This is the standard model for Unity. In pseudocode it's
ECS
Separate data from behavior and control behavior by the relationships of data. In this model Entity refers to a grouping of Components, Component is a structured piece of data, and System performs behaviors on the data. Here is some pseudocode
ECS can be more difficult to conceptualize than the others, but it tends to be a bit better suited to optimization both because of better data locality and because it is easier to parallelize. For an operation like the above it is easier to have all the positions and velocities tightly packed in memory to better utilize cache and the ram bus than a model where those bytes are interleaved with other stuff that isn't being used by that operation. It also becomes easier to parallelize because you can schedule Systems, which have clearly defined data dependencies, relative to each other. If you call update on a GameObject in either earlier model almost anything could happen to any object so it becomes unsafe to update them in parallel (ex. player input could trigger state changes in an interactable), but in Systems you know what's happening.
I'd normally go deeper and discuss memory layout, scheduling, and more but this is already kind of long for a comment written on my phone.
Conclusion
Personally, I don't know that I would recommend ECS to someone new to the domain. Compose is a whole lot easier and will work just fine for the majority of projects. But, ECS is good for making people think more about their hardware and how data is modeled when not trying to jam everything into OOP.
At my job I work fairly closely with HSMs and write software to interface with those in a cross-platform manner. Specifically, we use libraries that implement the PKCS #11 standard to generate and manage cryptographic keys on said HSMs. Because I don't have a physical HSM at my disposal, I use software PKCS11 implementation libraries that essentially emulate the behavior of an HSM on my local machine. On my team we use Mozilla's NSS, but another popular software implementation is OpenDNSSEC's SoftHSM.
I thought it would be a fun challenge over the winter break to start writing my own toy PKCS11 software implementation. I haven't even gotten to the actual cryptography yet (which I plan to implement with Botan). It's been mostly working with sessions and tokens and objects and representing those on the disk as needed. Lots and lots of reading through the specification. It's been really great because it's made me significantly more familiar with the standard which I imagine will be helpful in my day-to-day goings on at work and debugging various parts of the application. I'm looking forward to doing the actual cryptographic operations.
Once I make more meaningful progress I will un-private the repository so everyone can make fun of my C++ code.
For posterity: HSM = Hardware security module.