Passing question about LLMs and the Tech Singularity
I am currently reading my way thru Ted Chiang's guest column in the New Yorker, about why the predicted AI/Tech Singularity will probably never happen (https://www.newyorker.com/culture/annals-of-inquiry/why-computers-wont-make-themselves-smarter). ETA: I just noticed that article is almost 5 years old; the piece is still relevant, but worth noting.
Good read. Still reading, but so far, I find I disagree with his explicit arguments, but at the same time, he is also brushing up very closely to my own reasoning for why "it" might never happen. Regardless, it is thought-provoking.
But, I had a passing thought during the reading.
People who actually use LLMs like Claude Code to help write software, and/or, who pay close attention to LLMs' coding capabilities ... has anyone actually started experimenting with asking Claude Code or other LLMs that are designed for programming, to look at their own source code and help to improve it?
In other words, are we (the humans) already starting to use LLMs to improve their code faster than we humans alone could do?
Wouldn't this be the actual start of the predicted "intelligence explosion"?
Edit to add: To clarify, I am not (necessarily) suggesting that LLMs -- this particular round of AI -- will actually advance to become some kind of true supra-human AGI ... I am only suggesting that they may be the first real tool we've built (beyond Moore's Law itself) that might legitimately speed up the rate at which we approach the Singularity (whatever that ends up meaning).
The problem with that is LLMs don't have source code in the way that would be meaningful here. You have a program that provides an interface, of course, but the "running the LLM" part of that program consists of loading a really big array of numbers and doing math on them. The array is just data, not code, but that's the part where the "thinking" happens, to the extent these things think.
You can (and people definitely do) use the LLM to generate increasingly fancy wrappers around this process, but it doesn't make them smarter.
Noting my edit to my original post ... I get your point, 100%. Nonetheless, if it helps us move faster in the process of testing and experimenting in how to make LLMs smarter, isn't this still a leg up in the process?
Of course, I do not expect LLMs play any non-trivial role in the actual training process, which is where the "magic" is happening, and offhand, I don't see how they ever could. Perhaps those programmatic "wrappers" can only do so much, can only be optimized so much, and the LLM contribution to that work is also, ultimately, trivial.
IDK. Like I said, just a passing thought.
It could potentially help the actual researchers save time, sure, but that's drastically different from the LLM actually improving itself. There is no evidence that current state-of-the-art LLMs can do anything even remotely close to the process of improving an ML model's architecture or training process in novel ways, which is what would be required for the type of "self-improvement" described. LLMs aren't even good at coming up with novel solutions for much simpler software engineering problems -- they very much rely on existing methods and information in their training data.
I agree with @em-dash 's assessment with one other thing to add: a fundamental limitation of LLMs is context window. Once you reach this limit, the AI starts to forget things from earlier in the conversation.
Right now, the best I've seen is a million tokens, which is a lot, but isn't enough to hold larger code bases. There are some hacks, like breaking a problem down and dispatching the parts to sub-agents. But the context window places limits on how big a problem the LLM can take on.
Having worked with most of the high end models, I am confident that they are not effective for much without either significant guardrails or human oversight. By which I mean, somebody has to say, "yes that's what I wanted" or "no, it's wrong because" at the end of the process. Something like a test suite could replace that oversight, but then you've just moved the hard problem to writing bigger and more complex test suites.
Anthropic employees use Claude to code Claude: https://www.anthropic.com/research/how-ai-is-transforming-work-at-anthropic. The study indicates that it's increasing code output, but I'd be very skeptical, not only because it's from Anthropic but because the data is self-reported. More credible studies have demonstrated developers being less productive with AI agents like Claude Code, but I'm sure it depends on the person. Anecdotally, I mainly use Claude Code to write boilerplate, and I'm fairly confident it saves me more time than I spend writing the prompts and extra time debugging, because the boilerplate is easy to verify.
This means we may be getting to the singularity faster, but the curve isn't steep at least yet. You can't ask an LLM to improve itself 1000 times to get a super-LLM: there may be 1000 minor fixes and niche optimizations, but we almost definitely need at least one more breakthrough to get from a top-tier model (e.g. Opus 4.5) to human-level AGI, and current LLMs aren't discovering breakthroughs.
Well, actually...in a sort-of breakthrough, Erdos problem #728 was solved "more or less" fully autonomously by AI using a novel technique. The significance is discussed in the linked thread, most of which goes over my head, but my understanding is the AI still needed human assistance. More generally, cutting-edge researchers like Terrace Tao are using LLMs and reporting benefits, but (like with code) I haven't seen an explosion in discoveries, because the LLMs still need human guidance.
And something you may be interested in: what would happen if you asked Claude Code to improve a codebase 200 times? Someone did that and wrote about it. It's a short blog post, so rather than restating I recommend you read it. The tl;dr and relevance here is described well by its last sentence: "oh and the app still works, there's no new features, and just a few new bugs"
Honestly, I think this also overlooks that improving the actual code in terms of fixes or optimizations is not necessarily improving the LLM, definitely not in the sense implies by the whole singularity thing. The LLM isn't the Python code, it's the model. You can make myriad improvements to the app without ever making a single improvement to the model, and for the singularity to be a thing, an AI model would need to be improving the actual model itself. Claude could rewrite its entire codebase without ever doing that.
Models like this are improved by changing their architecture or how you train them (or what data you're training them on, but this is generally not what the singularity people are thinking of). You can make the actual code for Claude better from a software engineering perspective without remotely changing the underlying architecture of the model, much less the training approach. While LLMs can write the Python code for implementing existing approaches and architectures, especially in small/simple situations without a large codebase to interact with, there's no evidence of them even making incremental improvements on actually coming up with novel approaches to architecture or training that lead to improved outcomes -- and if there were, we'd be hearing about it.
The Erdos problem thing is cool but does not actually bear that much resemblance at all to what an AI would have to do to actually meaningfully improve itself (and the linked github at the end of that thread gives a lot of caveats in regards to AI contributions to Erdos problems).
I have so many feelings about this, where to start.
First of all, I think that we are still a ways off of any intelligence explosion, whether that be in humans or machines.
Using LLM's can increase productivity, but it is both my experience, and my general understanding, that it actually makes the user slightly dumber, not smarter, when it is used. It's easy, solves problems fast, requirering little critical thought. If what it does is something you were already doing, then you do it less often, and get progressively worse at it. If what it does is something you've never tried doing, then you never acquire that skill.
I suppose that there could be an argument for an aggregate increase in intelligence between the user and the LLM. That the human might get slightly dumber, but that the LLM gets much smarter, and thus the total intelligence has increased. The only way I see to evaluate that is by comparing the output of the combined Human/LLM system as opposed to the unassisted Human. But, when I do that, I again see only a decrease in quality (at least in my own field of software engineering).
The electronic systems that we cureently call AI (which are complicated beasts of many disparate parts) are also far FAR from any autonomic recursive increase in intelligence. Letting a model LLM loose, letting it attempt to modify and upgrade it's own infrastructure would have laughably catastrophocic consequences. LLMs can certainly generate massive amounts of functional code in the blink of an eye. But it is fragile code, fully functional in the narrowest cases, a buggy mess in the face of anything unexpected, a tapestry of the front page results of a myriad quick google searches. It has to be vetted by humans still, has to be padded with the missing pieces, has to be corrected, if you want anything sound. Cettainly if you want anything robust enough to recursively rewrite itself.
Second of all, we've build so many tools throughout our history that has helped us compute things better. Like, the current computer that you are probably reading this on is an overstuffed toolshed of things that make it easier to compute things, one tool holding up the other, in a chain of bootstraps that let's us intepret the consequences of unique spatial and temporal patterns of electrons as images, sounds, buttons and letters from friends and strangers. So many people have contributed to just the code running on your computer, each one solving a different problem, with a different scope, dependent on other peoples solutions and fascilitating other solutions to other problems in turn. And, that computer is only able to run that code because an absolutely staggering amount of other computers are already running, ready to interconnect with any new computer, filled with the tools we need to set up and continue development. And then there are all the people that have thought up and rigouresly proven the mathematical principles that are fundamental to our computing. And the engineers that have build the countless electronic and mechanical machines needed to keep computers being a thing.
It is baffling to me that we are currently so fascinated by computers that can get smarter by themselves that we forget that all of human invention, ingenuity and intelligence is the consequence of years upon years of collaboration and private triumphs. It saddens me that you can only mention Moore's Law (which, while an acute observation by Moore, is not much of a tool) and large language models out of all the steps that we've already taken - and are still taking - towards furthering our understanding of our world.
And, thirdly and finally, what the fuck is up with this modern belief in AI singularity? Why do we believe that intelligence could ever be a runaway effect? That it is even plausible that anything could just recursively learn ad infinitum without limit? Have we all gotten so comfortable with the supposed singularities in black holes, that we've forgotten that singularities are a logical problem? That when the result of any postulate, any calculation, any thought, is a singularity, then one should probably reconsider? I am unconvinced that any form of intelligence singularity is likely or desireable. It seems to me too clean, too simple. It reeks of deification, of over-anthropomorphication. it sounds like the screams of sacrifices to volcano gods that could have gone on to study plate techtonics.
What is the point of this "singularity", what do we hope to gain? Why in all nine hells do we doggedly pursue it, seemingly without question?
My mind? Boggled.
Marketing - that's why. It's a central tent pole in the never-ending tech bro scam cycle. They are selling Roko's basilisk.
As for my own intuition on the matter... I tend to toss out the math and look at hardware. We know for a fact that consciousness, whatever it is, however it works, comes to an instant stop the moment you shut down the micro-tubules using anesthesia. We also know for a fact that this works just as well in a single cell organism as it does in any other living thing. There are no living things without these micro-tubules as part of their structure, it's in every cell. That means evolution itself has selected for them so strongly that we have no examples of life or consciousness or sapience without them.
There are no micro-tubules present in a computer. Further there is also no stand-in present for the quantum effects these micro-tubules generate, which is provably required to be present for consciousness. By this reasoning, there are no conscious machines, yet - and we wouldn't even expect to see any become possible until we are running light-based hardware that is designed to perform the same functions we see in living things.
Evolution has had several billion years of uninterrupted uptime to perform field testing more brutal than anything humans have ever done with code. I think it's a bit premature to assume that evolution itself hasn't already begun to approach some hardware optimization limits that machines wouldn't be able to push past with impunity. Nature may already be nearing those limits. Something better might be possible but we're sure not going to figure it out until after we answer the how and why of biological consciousness.
We've invented syncopathic librarians that are more convenient than search engines in some circumstances, and are suitable as a pocket grad-student in others. Too bad people still need to check all of their work all of the time. It might be enough to make robots about as useful as a teenager at doing the chores. If you want more than that out of it, you've got to specialize the diet you feed to them during training, that's still 'narrow'. It's a very useful tool that can be embedded into just about every other tool... but that's it. Something else is still missing just to bring it up to our level.
The current tech bubble will continue, it seems, until they manage to stuff the entire corpus of human knowledge into the context window, or they run out of funding. Meanwhile, all the AI power you'll likely need in anything you're doing yourself will come free, bundled in your next graphics card upgrade. I'd give it a couple years tops before someone figures out a way to train them better on those cards and on small (even tiny or micro) data sets. That's the point where people finally realize the data center model we're rushing now is a scam.
As an aside, I know it's just a typo, but I love your new word.
TLDR; I don’t expect a singularity any time soon.
There are two things to remember when you hear about the current and future capabilities of AI:
I’ve been using Claude code and copilot a lot recently and everything I’ve done follows this pattern:
There’s a good comic circulating about AI.
In the first panel, the person asks “is this mushroom edible?” And the AI answers “yes”.
In the second panel, the person is very ill and lying in a hospital bed. And the AI says “You're right, it's a poisonous mushroom. Would you like to learn more about poisonous mushrooms?”
As others noted, LLMs don't exactly have "code". However, LLMs have been used as a part of the LLM training process for some time. In that sense, yes absolutely.
In another sense: the agent harness, which is an increasingly important part of how effective LLM powered agents are at real world tasks, also yes. The big model companies use coding agents extensively when creating the harness and scaffolding, which shows in the sheer volume of bugs the harnesses have.
I didn't read the article you linked so I may not have all the nuance about what intelligence explosion refers to. But in terms of practical application of AI agents at real world tasks we are undoubtedly in the midst of an intelligence explosion.
In terms of actual intelligence, as we might define it, LLMs arguably have none at all. It depends on how you frame it. They provide an illusion of intelligence that is so good they can do intelligent things, which is to say things that could previously only be accomplished with human intelligence. But it's achieved, essentially, through advanced pattern matching rather than anything that could be described as understanding. It's hard to imagine a path from there to true AGI but at the same time it's not difficult to imagine that at some point the illusion of intelligence could get so good that it's practically the same as the real thing for many applications.
It's all so weird.
The article is a side-note. The "intelligence explosion" is the idea that an AI is, first of all, unarguably "intelligent", and in fact intelligent enough to be able to improve it's own code base better/faster than a human could.
We are definitely not there yet, and in fact, I still don't think LLMs will ever get there, not w/o at least a couple more paradigm-shifting breakthroughs by humans, first.
But I am beginning to think that, indirectly, the LLMs may be to the point of helping humans reach those breakthroughs faster than we would, w/o them.
Ah, yes in that context we're nowhere near an explosion. Or at least the existing technology doesn't put us near one, who knows if there will be breakthroughs in the near future.
Yes LLMs are already helping move the technology along faster than humans alone could do it. I don't think there's any doubt of that. The only question is if the path leads to the vicinity of AGI, which I think is safe to answer yes. It doesn't matter of LLMs themselves will have anything to do with AGI, they will definitely accelerate many aspects of technological advancement and some of them will contribute to eventual AGI.
I'm pretty confident that LLMs have already plateaued and will see little substantial improvement in the next 10 years. And by substantial, I specifically mean "things that feel like sci-fi". They'll become more useful and more efficient within the same paradigm, but there won't be any earth-shattering developments.
If AGI ever happens, I am not convinced it'll have much to do with LLMs. LLMs are fancy probability engines. They are to me closer to the Mechanical Turk than to true intelligence.
That said, I'm a science fiction guy who reads about some of that stuff. Not a STEM person.