16
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
been learning electronics fundamentals! I've always been a computer nerd, but I'm 100% self-taught. it's been a near-lifelong frustration of mine that I can't 100% conceptualize how a computer works... like, on the physical, electron level. how, to HUGELY simplify, a series of billions of "yes/no" options can have complicated logic applied to it to eventually give us what we see in front of us on this website. it's a "black box" of magic and mystery but I NEED TO KNOW.
over the years the "black box" grows smaller and less scary as I have passively and casually learned more, and after a recent impromptu lesson from my partner (who has a PhD in this stuff - lucky me!), I think I'm starting to finally "get it" and I am determined to demystify the magic at long last. my partner had a bit of a lightbulb moment in my gaps of understanding, and we built a simple flip-flop together. it's a concept I hadn't encountered before in my endeavors to teach myself how all this stuff works and it really opened my mind and is helping to make much more sense out of everything.
You've probably seen this, but for anyone else: https://www.nand2tetris.org/
It takes you from the simple logic gate through a simple computer. I haven't tried it but I've heard good things.
I had seen it, years ago, but forgot all about it - thank you for the reminder!!
Watching my partner work on building a LC-3 CPU from scratch visually through this course has similarly been enlightening for me, even after spending a long time working at both the higher & lower levels of abstraction
I don't think I'd be willing to personally work through this course myself, but all the materials are there, and I found the simulator used there much easier to understand than that one time I tried doing HDL.
I have also recently dove into "fundamental electronics", I've worked in software for almost 2 decades now but electronics wise I barely knew anything past those alligator clip/incandescent bulb/9V battery circuits we built in grade 7.
What got me really going is Ben Eater's Youtube videos. He has a series on basic digital electronics I was able to really pick up and understand - it helped to prototype the circuits on a breadboard as I went. I enjoyed it so much I picked up his "build a 6502 computer on a breadboard" kit and have been slowly plodding away at that in my free time for the past ~year. The videos for the 6502 project are probably the most information dense on Youtube (each 30 minute video takes me a couple weeks to get through) but it is some of the most clear and rewarding technical instruction I've ever consumed.
edit: The playlist above is good but they weren't the videos I was thinking of, it was these buried in his 8-bit computer playlist: SR latch, D latch, D flip-flop - grokking these really levelled me up.
I've been learning to reverse engineer games with ghidra. I feel like I have a pretty good grasp on it with older pc games. Though I have hit a wall now, attempting to reverse a ps2 game. I just can't find a good way to debug pcsx2 games. At least not in a way that can be easily translated back to the decompiled code in ghidra. The romhacking forum posts I've found just amount to "use cheat engine and ghidra" with no further explanation. If anybody has any knowledge on reverse engineering ps2 stuff I'd appreciate it.
Otherwise I will continue to bash my head against it until something starts to work :)
I'm not familiar at all with PS2 games but have just a tiny bit of familiarity with Ghidra. Is the problem here that the PS2 ISA is not supported in Ghidra?
My understanding, from working a little bit with the Xtensa ISA in Ghidra, is that Ghidra is designed so that you can make a plugin for a new ISA and it should Just Work with the rest of the system & decompiler.
I have a ghidra extension that supports the MIPS instruction set on the ps2. That's all working fine. I'm just limited in what I can do with it if I can't attach a debugger to the game with the original memory addresses accessible. The emulator remaps all the memory and I think I'm just misunderstanding how to debug emulated games.
I know people have used cheat engine to access the emulated memory, but I can't figure that out apparently.
cool! I've always wanted to learn some reverse engineering, but it really does feel like a scratch-your-own-itch kind of thing - if you don't have a burning need to crack that 24 year old abandonware it can be hard to know where to begin. Any recommendations on how to get started?
I started by following along with a YouTube tutorial on reversing roller coaster tycoon. I already have a lot experience with high level languages and at least a surface understanding of assembly languages. After a couple hours of tutorials on specific software it gave me enough tools to go off on my own. It's definitely a slow and tedious process and I have pretty good motivation pushing me through it.
I've been tinkering with my StarCraft 2 bot. At this point I've pretty much given up hope of ever creating something competitive for the SC2AI ladder, but it is a fun vehicle for playing with interesting algorithms under real-time constraints.
This week I've implemented a method that enables the bot to select its actions using a neural network evolved by Grammatical Evolution, a variant of genetic programming that employs a pre-defined grammar to evolve whatever the grammar describes.
For example, you could have it evolve valid JSON documents or even Python programs if you specified a grammar complex enough for that. It's a very flexible method.
In my case the grammar prescribes a simple expression language that, when evaluated, results in a neural network.
The evolved output of the grammar looks like this:
This represents a single neuron using a single input connection. The indices 977 and 333 are taken modulo the actual number of inputs/outputs, and RELU6 is the RELU6 activation function,
max(min(6, x), 0).The expression is converted into Reverse Polish notation using the Shunting yard algorithm:
Having it in RPN means that the expression is very easy to evaluate from left to right using just a stack.
Multiple neurons are evolved and combined into a neural network. The network runs every three seconds of in-game time and scores each possible option the bot can take (e.g. build another barracks, research a weapon upgrade, do nothing, etc.).
The genetic algorithm driving the process aims to minimize or maximize randomized goals using a method called Lexicase selection. The goals I use are selected from the set of all possible objectives of the form "make as many/few units of type X as possible while still winning against the built-in AI".
This method results in a diverse population (which is crucial, because I need to use small populations -- playing a single game takes a few minutes) and is supposed to also do well on the actual objective (i.e. winning the game).
I probably should teach the bot to use other units than the basic Terran infantry, but that's for another week...
A no markup (or extremely small markup) domain registrar, with a large portion of the profits going to a charity of your choice.
Is anybody here interested in that?
Don’t you need to be in a sort of in-group to have a domain registrar? Like be registered with ICAAN? Or just pay another registrar for domains and add some markup.
I think that’s a great idea, but I don’t see how you’d achieve it (and I’m not talking about implementation).
I found a website which gives registrar prices + 5%, given you have a subscription. So I think the key is making sure to sell like >100 domains a year, just a quantity game in the end.
I just bought a domain for my personal page.
I like the idea and I'd support it. How would it work without turning in profits though?
Profits are important (I like being able to eat), but I feel like as long as upkeep is paid for, is be happy (especially with the positive impact this would have).
What do you think a decent pricing model would be?
Nothing too technical.
Wrapping my head around Azure really. Fundamentally different to AWS, but feels much more like the grown up in the cloud game. Despite its drawbacks.
I’ve been studying and designing how to set up a reverse proxy with let’s encrypt ssl certs so I can expose a lot of my self-hosted services. I’m currently using cloudflare tunnels, but with Plex breaking the terms of service, I’d rather have Cloudflare just handle my DNS, and I handle the rest. The problem is that I’m super afraid to even set it up because of my flat network in my house. I have no segregation of any type and just a few different wireless networks for IoT devices who can’t connect to 5 GHz. It seems like if I open up ports I’ll need to do a ground-up overhaul of how my network and infrastructure is set up.
When you say reverse proxy, shall I assume you have a remote server as well?
Nope, no remote server just hardware at home. The end goal is that I want to have a nice home lab that I can access with little to no ports being opened up.
I've been working on a new boardgame project with my brother at new.boardzilla.io. There is something kind deeply fulfilling about working on a passion project with a family member. I recently watch The Sparks Brothers (great film btw) and thinking about the sorts of deep artistic relationships you can have with someone over time. I guess this is a bit of an outpouring from that.
I'm slowly teaching myself Rust by writing a game that runs on a Raspberry Pi Zero with an Adafruit OLED bonnet for I/O and a PiSugar 3 battery. My idea (right now, it's changed a few times already though) is an idle-type thing where a you help guide a pocket-pet space captain to explore a seed-generated galaxy, fight pirates, find profitable trade routes, and other various tasks (think if Tamagotchi and EVE Online had a baby).
Still very early stages, and no real end (or beginning, really) in sight, but I have vague lofty ambitions such as using the RTC on the PiSugar to support offline progression, and shared galaxies with a server component that would allow pets to influence and interact with one another in some way. I also want to design a simple 3D-printable enclosure for the Pi/OLED/battery combo, probably using OpenSCAD.
It's highly likely that I'll abandon this before it ever gets anywhere interesting, and even if it does get somewhere interesting I wouldn't expect anyone to play it other than me, but so far learning Rust is quite neat and I'm having fun which is my main goal for now.
I've been overhauling my media library software to use
natsortinstead of a very badly performing implementation which would keep querying a SQLITE table and cut off small sections of the end the file path until it had multiple matches... as you can probably imagine there were many problems with thisI also discovered a very good MCDA python library and I used it to add a feature to do automatic MCDA analysis and sorting based on multiple-criteria.
For example (the CLI output is markdown):
Shape
(1360, 4)
Goals
Maximize
Minimize
yes... that you weren't using binary search instead 😱
but really, I'm curious about how you incorporated
natsort. I didn't know about that library, but it seems very useful. When I use it, I getwhich is a great way of doing this key (and not what I expected), but at the same time I don't see how this can be indexed given SQLite's limitations.
sqlite3_create_collation? But I don't think Python supports this.Sure, here is the relevant code. I don't use natsort with SQLITE indexes directly but I use natsort after fetching some data.
The previous function get_ordinal_media would also query the database for more data. This is useful when using
--randomwhich will initially only select a random subset of the table based on rowid. Last night, I replicated the fetching functionality of get_ordinal_media in the get_sibling_media function.So now I can do
library watch --random --fetch-siblings eachand it will get a random list of files and then get the first result from each parent path substring search. There are probably better ways of doing this, like with a correlated subquery but it feels more flexible as-is.Here is the relevant documentation:
Play media in order (similarly named episodes):
The default sort value is 'natural_ps' which means media will be sorted by parent path
and then stem in a natural way (using the integer values within the path). But there are many other options:
Options:
Algorithms:
Values:
Use this format: algorithm, value, algorithm_value, or option_algorithm_value.
For example:
library watch -O human
library watch -O title
library watch -O human_title
library watch -O reverse_compat_human_title
library watch -O path # path algorithm and parent, stem values (path_ps)
library watch -O path_path # path algorithm and path values
Also, if you are using --random you need to fetch sibling media to play the media in order:
If searching by a specific subpath it may be preferable to just sort by path instead
library watch d/planet.earth.2024/ -u path
library watch --related # Similar to -O but uses fts to find similar content
library watch -R # equivalent
library watch -RR # above, plus ignores most filters
library watch --cluster # cluster-sort to put similar-named paths closer together
library watch -C # equivalent
library watch --big-dirs # Recommended to use with --duration or --depth filters; see
lb big-dirs -hfor more infolibrary watch -B # equivalent
All of these options can be used together but it will be a bit slow and the results might be mid-tier
as multiple different algorithms create a muddied signal (too many cooks in the kitchen):
You can even sort the items within each cluster by auto-MCDA ~LOL~
https://github.com/chapmanjacobd/library#media-database-subcommands
I created a wishlist website using Flask! Super proud of it. Was able to lean hard on ChatGPT4 and had no hallucinations or fake libraries and what followed best practice advice.
I wrote a whole bunch of documentation for my (never finished, but fully functioning) self-hosted Discord bot. It only took 7 years...
Been tinkering with the rust crate lopdf to construct a very basics pdf file with just a png image and some non-ascii encoded text. Super fun to get just a basic page with some content on, looking at the raw file in vim to verify the representation is as expected. I want to build on this to make a basic library for creating pdfs compact pdfs based on some structured layout/content data.
I've challenged myself to use Samsung DeX (on my Fold 4, with a Dell portable monitor and a Logitech Bluetooth keyboard) for the entire month of December as my only computing device.
So far, I kinda like it. Termux has been great, Python and Rust are natively supported so I've set up VS Code Server as my IDE (sadly I can't get all the extensions I use on PC, but it's fine), and by using PRoot I've set up a Debian instance so I can run Ardour and REAPER. I theoretically can even kinda run Windows stuff using Winlator, but I haven't tried it yet and I expect there to be crashes, and lots of them.
It doesn't seem like there's a huge roadblock that would prevent me from using this forever, but we'll see how the month progresses.
I just wish NexDock made a lapdock with a better touchpad, so I would just have to carry the it and the phone instead of 3, potentially 4 (if I also bring a Bluetooth mouse) devices with me. I might still end up buying the NexDock 360 Wireless as a Christmas gift for myself, though, if I do decide to stay on DeX.