What is the most difficult or complex thing you have programmed, created, or otherwise made?

I once wrote a program in Mathematica to find all the best solutions of a rectilinear Steiner tree on nine points out of spite. It took two days to write the code for, then another 42 hours to execute in parallel across nine machines. It was for a class I had to take with a professor I had taken issue with in the past for many reasons - this particular time he insisted that the computer science people were somehow different from the math people in the class. This guy also ran a Programming in Mathematics class that was hostile to his students who knew more than him about programming (though I never took it, in an effort to avoid him); I was fed up with this and years of other idiocy from him, and decided that since this was the final project I’d pick something difficult to solve, write something truly eldritch, and leave the following Dijkstra quote on his review:

Programming is one of the most difficult branches of applied mathematics; the poorer mathematicians had better remain pure mathematicians.

One of the more petty things I’ve done like this, but that guy was awful. I did get a 110/100 on the project though.

Edit: To give an example of some of the “other idiocy,” I once tried to take a Real Analysis class this professor was teaching and dropped it because he kept marking my proofs wrong and making me go to his office hours and explain them. I decided to try retaking it the next semester (even though I had already completed a Complex Analysis class at this point) and found he was using some of my proofs in his distributed homework solutions.

Second edit: This is admittedly not the most difficult or complex thing I’ve made, but is the only one with both a problem that’s easy to understand and with a story behind it.

The most complex thing I've ever worked on was as part of a larger team of developers. In medical science there is a device that's been around since the 50s called a flow cytometer. It uses lasers and photodetectors to count cells. With the help of specialized reagents that bind to specific cells, it can count say, the number of a particular sub-type of immune cells in a blood sample to provide diagnostic data for a particular disease.

For about 4 years I worked on a team that wrote the desktop software that ran a newer cytometer that was about the size of a bread box (previous models from competitors where the size of refrigerators and cost 100s of thousands). It both ran the device, collected the live data stream, plotted graphs of that data, and ran preconfigured analysis based on the type of cells being analyzed.

The codebase was fairly large, somewhere around half a million lines, thousands of classes, and was very strictly test-driven with over 35,000 unit tests. Running the full test suite took about 45 minutes.

I came in near the end of the project, and got to help write some key features. Some of them driven by some complex statistical math that I had to learn practically from scratch (with help). The device+software was submitted for FDA approval and eventually was approved. I have a small engraved acrylic block that we all got thanking us for our efforts, I'm very proud of it.

Small side story, my favorite thing that I did as part of that project was actually a joke. The data format used in cytometry is an open standard. So at one point I wrote a small Java program that would take an arbitrary image file, and re-encode it as cytometer data. I then took a picture of the project owner, encoded it, and made sure they loaded it during the next project demo. Got a good chuckle from everyone.

The most complex thing I’ve probably made as a self-taught programmer is my operating system, duckOS. It’s made with a mix of mostly assembly, C, and C++, runs in 32-bit protected mode, has preemptive multitasking, memory management, etc. It aims to be UNIX-like, and has a userspace window manager and compositor called pond. I’m currently working on ironing out some of the instability and bugs with memory management and shared memory, but I recently took a break from that to make a small GUI-based connect four game for it.

I can think of 2 things that were hard early in my career. I don't think they'd be hard today because of the better tools we have, and for me because of the additional experience I've gained over the years, but they were both learning experiences.

In terms of debugging, the difficult debugging problems are generally race conditions of one sort or another. Or any other problem that changes under use of the debugger. You get a problem to happen. If you're lucky it happens consistently, but usually it doesn't. And then you try it out with the debugger and it doesn't happen. This was one of those bugs.

I remember working on some sort of driver back when I worked at a company that did contract development. I don't recall whether it was a network driver, or some sort of USB thing. But there was a problem that we couldn't consistently reproduce, but happened sporadically. This would have been late 90s on MacOS 8 or 9 at the time. The dev tools were OK, but the OS didn't have memory protection so any program that messed up could potentially take the whole machine down with it. On those rare occasions it did happen, the machine was so hosed I couldn't use the debugger to see what went wrong. After a week of adding logging, and manually stepping through the code in the debugger and not seeing anything obvious happen, I almost threw up my hands and gave up. I read over the code (which a more skilled engineer than me had written) to see if there was anything obviously wrong, but nothing seemed out of place to me. (But I didn't really know what I was doing. Device drivers are absolutely out of my wheelhouse.) Then a thought struck me. There was a variable being used, but I couldn't figure out what it would be set to by examining the code. After working through all of the possible paths that you could take in this 1 function, I realized that the variable wasn't being initialized in all cases, and would sometimes end up with an essentially random value. I added an assignment where it was declared and the problem went away. If I recall correctly, the problem never occurred while running the debugger because the debugger inserted some extra initialization (like zeroing certain areas of memory, or something), making the problem disappear. Nowadays, my compiler would catch an uninitialized variable the first time I built my new code, but for whatever reason, that compiler didn't (or we had so many warnings, we ignored it). So dumb and such a waste of time.

The other one also from that era is an optimization problem. I was working at a company that wrote software for print shops. We did a lot of Photoshop plug-ins and various types of file format conversions and outputting a lot of PostScript. (Ugh, so much horrible PostScript.) We had a client that had a huge offset printer and they wanted to send losslessly compressed files to it. Their test file was a 120,000 pixel- wide 1-bit per channel CMYK file that they wanted compressed with what is essentially the same compression that fax machines use. (It's a fascinating compression. It's lossless, but compresses on average about 10:1 because it's very use-specific.) This was in the era when most desktop computers had around 64MB of RAM. I wrote up a straightforward version of the code and tried it out. It took 15 minutes to compress the huge test file. That obviously wouldn't work because they sent dozens of these things to the printer per day. This was the project where I really learned to use a profiler for the first time. I would run the test image through the program and see which functions took the longest. Then I'd optimize those functions and run it again. Little-by-little I whittled it down. Some stuff I had to completely rewrite. Other stuff I just needed to mark as being inline to remove a function call. Eventually, after several days of this, I got it compressing the file in 15 seconds. So a 60:1 improvement in speed. That was a lot of work, but it was worth it!

I made a brainfuck just in time compiler (https://github.com/danthedaniel/bf-jit) without using any jit-specific libraries. The back-end includes hand written x86-64 machine code. I don’t know shit about x86 but I learned just enough to make the project work.

It was fun, but really really difficult to debug. At one point I needed to use a reverse able debugger, rr. There’s actually one bug left related to reading from stdin from within just in time compiled code. I’ve been putting off fixing that.

My favorite parts of these barely in reach projects are the moments of epiphany when I finally realize what’s broken with a development build. I’ll be staring at the paused debugger screen, stepping through instructions, and recall something I must have read or heard in passing years ago about caller vs. callee saved registers. Then it’s just minutes until I implement a fix and get the payoff I’ve been waiting weeks for.

I once invested more than 600 hours in one factorio save. I enjoyed every minute of it. Never before have I worked on something that long and intensly as I usually get bored fast.
It was a beautifully organic looking factory with hundreds of trains tousands if not millions of belts :-) every item was automated, used every kind of powersource, used every sort of defense, defense and repair itself was automated. most of the things where nicely decorated, but there was not really any order to it, similar to how vaclav is building his factories, lots of trains and belts all over the place.

My hardest project is the one I'm currently working on as a grad student.

A big chunk of TA time for the first month of every semester is spent getting students up and running with their IDEs on lab machines...and then, in turn, on their laptops. Even given the most detailed step-by-step instructions with screenshots in the world, a good chunk of them will still manage to mess it up. To reduce the time spent on basic setup tasks, some professors have been using AWS Cloud9, a hosted IDE, since you only have to set that up once and can access it from any device and the environment will remain the same.

The problem is that some professors have been telling students to make their own AWS accounts, while others have been creating accounts for students under their own accounts, and that runs the risk of unexpected charges since professors are not usually good with AWS permissions and students will sometimes create multiple workspaces and use up their free tier on their own accounts.

So I have been tasked with creating a system to automate the creation of these environments in a way that is both secure and cost-effective, with the end goal of making this available to a much larger audience (i.e. other universities).

We decided to use another web IDE due to limitations of AWS Cloud9 workspace setup on existing EC2 instances, and I am really just laying the groundwork with tasks like setting up LDAP and automating processes to provision infrastructure.

When I actually think about what I'm doing, I'm basically going to be writing an infrastructure manager like K8s and proxying around a fair amount of traffic, and securing this is of huge importance since giving potentially hundreds of users shell access guarantees people will be testing the boundaries of the system. A couple security-minded people have given my plan for locking it all down a thumbs up, but it feels like I'm not even going to finish this before I graduate. I know how long big systems take to come together from my time at Oracle on a team with some very smart individuals, I will be surprised if I manage to accomplish all my goals before I graduate.

Two things come to mind, one from my professional life and one that was a college project.

At $dayjob we use RabbitMQ pretty extensively.

I wrote a little library that our various Python services use as a connection pool.

Sounds easy, right? Connection pooling is a common and well-understood problem.

Well...most of the time with connection pooling, a connection is almost entirely stateless (such as a Keep-Alive HTTP 1.1 connection) or has pretty simple state (such as a cache of which SQL queries have been already been sent to the database server as prepared statements).

RabbitMQ (more specifically, the AMQP protocol that it implements) has very stateful connections, which makes pooling and reusing them challenging.

It has connections, which are a TCP-level connection to the server. From a connection, you open a channel, which is the thing you do all your actual work on. It's common and idiomatic for an application to open only a single connection to a RabbitMQ broker, then multiplex many channels over that connection.

Once you have a channel, you're required to declare all of the resources you intend for that channel to use, before you use them. This is a really smart design on AMQP's part - if you have a large, multi-tenant message broker (which is what AMQP was designed to support) there are N server side objects (such as queues), but any given connection probably only cares about M of them (where M is at least one order of magnitude less than N). By requiring "you must declare you're going to use it before you can use it", they limit per-channel overhead to O(M) rather than O(N).

Oh, and RabbitMQ in real-world environments is run in a highly available / clustered setup (typically 3 or 5 nodes because those are magic numbers in distributed systems).

So the connection pool I wrote handles all of that. You have to tell it up-front which server-side resources you plan to use, like the underlying protocol requires, but other than that you can just ask this black box "give me a channel plz" and it magically handles reusing things if it can, opening new things if it can't, and reconnecting automatically if the network goes sideways or the server it's talking to restarts.

Behind the scenes - it's locating the 5 cluster nodes we run, picking 3 of them at random, then opening a connection to each one. It spins off a thread (really, an asyncio task) that does nothing but monitor the 3 connections and open new ones if they die for whatever reason (AMQP provides an underlying connection heartbeat that makes this really easy). Any time we do planned maintenance that involves this cluster, we restart one node at a time and wait a few minutes in between each restart, and the client services barely notice it happening.

When client code asks for a channel, it'll reuse a channel if there's already one open (but only if it has the appropriate resources declared). If not, it'll open a new channel (picking 1 of the 3 connections at random for load-balancing), declare the resources, then return it to the client.

The library is monitoring the connections in the background and automatically reopening them if they die, but there's always the chance of a race condition where the pool thinks a connection & channel are healthy, hands them back to a client, then the network dies, and the client code sees a connection error.

One of the promises this pool makes is that if the client sees a connection error like that, if it retries, the pool will have detected the error, and will be smart enough to use one of the other 2 connections it has already established (assuming the common case where a single RabbitMQ node is restarting rather than the whole system going sideways), reopen a new channel on that connection, re-declare the necessary resourecs, and on the first retry everything should "just work" like nothing was ever broken.

I'm going to write up the college project as a separate comment because this is already long enough...

This was a couple of years ago, and I don't remember what it was for, but I wrote an SQL query that was something like 80 lines long. I think it had 3 or 4 joins in it.

I started in industry out of college as self-taught programmer couple years ago, and mostly worked on small things.

Then last year I was handed over a feature implementation under guise of "the usual stuff". As I went working on it, it spanned and touched every part of our codebase and infrastructure, I had to make architectural decisions, manage multiple teams, maintain backwards compatibility in highly risk-averse environment while hacking up legacy codebase written by people who had long left.

I'm sure everyone here does it for breakfast, but that's the one I have done so far.

This was only following instructions provided by other people, but fixing a bug in my harddrive's firmware was fun.

https://www.overclock.net/threads/seagate-bricked-firmware-drive-fix-with-pics.457286/
https://sites.google.com/site/seagatefix/Home
https://msfn.org/board/topic/128807-the-solution-for-seagate-720011-hdds/
https://superuser.com/questions/365999/how-do-i-recover-data-from-my-presumably-dead-hard-disk