Passing question about LLMs and the Tech Singularity
I am currently reading my way thru Ted Chiang's guest column in the New Yorker, about why the predicted AI/Tech Singularity will probably never happen (https://www.newyorker.com/culture/annals-of-inquiry/why-computers-wont-make-themselves-smarter). ETA: I just noticed that article is almost 5 years old; the piece is still relevant, but worth noting.
Good read. Still reading, but so far, I find I disagree with his explicit arguments, but at the same time, he is also brushing up very closely to my own reasoning for why "it" might never happen. Regardless, it is thought-provoking.
But, I had a passing thought during the reading.
People who actually use LLMs like Claude Code to help write software, and/or, who pay close attention to LLMs' coding capabilities ... has anyone actually started experimenting with asking Claude Code or other LLMs that are designed for programming, to look at their own source code and help to improve it?
In other words, are we (the humans) already starting to use LLMs to improve their code faster than we humans alone could do?
Wouldn't this be the actual start of the predicted "intelligence explosion"?
Edit to add: To clarify, I am not (necessarily) suggesting that LLMs -- this particular round of AI -- will actually advance to become some kind of true supra-human AGI ... I am only suggesting that they may be the first real tool we've built (beyond Moore's Law itself) that might legitimately speed up the rate at which we approach the Singularity (whatever that ends up meaning).
The problem with that is LLMs don't have source code in the way that would be meaningful here. You have a program that provides an interface, of course, but the "running the LLM" part of that program consists of loading a really big array of numbers and doing math on them. The array is just data, not code, but that's the part where the "thinking" happens, to the extent these things think.
You can (and people definitely do) use the LLM to generate increasingly fancy wrappers around this process, but it doesn't make them smarter.
Noting my edit to my original post ... I get your point, 100%. Nonetheless, if it helps us move faster in the process of testing and experimenting in how to make LLMs smarter, isn't this still a leg up in the process?
Of course, I do not expect LLMs play any non-trivial role in the actual training process, which is where the "magic" is happening, and offhand, I don't see how they ever could. Perhaps those programmatic "wrappers" can only do so much, can only be optimized so much, and the LLM contribution to that work is also, ultimately, trivial.
IDK. Like I said, just a passing thought.
It could potentially help the actual researchers save time, sure, but that's drastically different from the LLM actually improving itself. There is no evidence that current state-of-the-art LLMs can do anything even remotely close to the process of improving an ML model's architecture or training process in novel ways, which is what would be required for the type of "self-improvement" described. LLMs aren't even good at coming up with novel solutions for much simpler software engineering problems -- they very much rely on existing methods and information in their training data.
You are confusing LLM's, which are predefined mathematical representations of human knowledge, with code, which runs pre-determined programs.
Currently LLMs have no agency, no desire, no drive. You (or LLMs) can write a bit of code to simulate this drive. But it will be very constrained and predefined.
Currently LLMs have really poor thinking skills. They learned how to write, how to draw, but not how to think. You can write code to teach an LLM how to think, but that is extremely constrained and predefined. It also requires the LLM to learn.
Currently LLMs can't learn anything new, on existing technology, without massive amounts of energy. This is what would need to change, I suspect, to achieve the AI Singularity.
Our brains are massively efficient at learning new things. And we have billions of brains. Yet we produce very few geniuses. To surpass us, LLMs would need massive efficiency gains or massive untapped energy sources.
Lastly, currently LLMs have no autonomy. It can't build new hardware or tap new sources of energy. All though, in spite of all his warnings about the risks of AI, Musk seems hell bent on giving AI autonomy :)
Although LLMs are definitely being used to further curate data and to generate synthetic training data which does directly influence the matrices that underpin their “intelligence”.
I have so many feelings about this, where to start.
First of all, I think that we are still a ways off of any intelligence explosion, whether that be in humans or machines.
Using LLM's can increase productivity, but it is both my experience, and my general understanding, that it actually makes the user slightly dumber, not smarter, when it is used. It's easy, solves problems fast, requirering little critical thought. If what it does is something you were already doing, then you do it less often, and get progressively worse at it. If what it does is something you've never tried doing, then you never acquire that skill.
I suppose that there could be an argument for an aggregate increase in intelligence between the user and the LLM. That the human might get slightly dumber, but that the LLM gets much smarter, and thus the total intelligence has increased. The only way I see to evaluate that is by comparing the output of the combined Human/LLM system as opposed to the unassisted Human. But, when I do that, I again see only a decrease in quality (at least in my own field of software engineering).
The electronic systems that we cureently call AI (which are complicated beasts of many disparate parts) are also far FAR from any autonomic recursive increase in intelligence. Letting a model LLM loose, letting it attempt to modify and upgrade it's own infrastructure would have laughably catastrophocic consequences. LLMs can certainly generate massive amounts of functional code in the blink of an eye. But it is fragile code, fully functional in the narrowest cases, a buggy mess in the face of anything unexpected, a tapestry of the front page results of a myriad quick google searches. It has to be vetted by humans still, has to be padded with the missing pieces, has to be corrected, if you want anything sound. Cettainly if you want anything robust enough to recursively rewrite itself.
Second of all, we've build so many tools throughout our history that has helped us compute things better. Like, the current computer that you are probably reading this on is an overstuffed toolshed of things that make it easier to compute things, one tool holding up the other, in a chain of bootstraps that let's us intepret the consequences of unique spatial and temporal patterns of electrons as images, sounds, buttons and letters from friends and strangers. So many people have contributed to just the code running on your computer, each one solving a different problem, with a different scope, dependent on other peoples solutions and fascilitating other solutions to other problems in turn. And, that computer is only able to run that code because an absolutely staggering amount of other computers are already running, ready to interconnect with any new computer, filled with the tools we need to set up and continue development. And then there are all the people that have thought up and rigouresly proven the mathematical principles that are fundamental to our computing. And the engineers that have build the countless electronic and mechanical machines needed to keep computers being a thing.
It is baffling to me that we are currently so fascinated by computers that can get smarter by themselves that we forget that all of human invention, ingenuity and intelligence is the consequence of years upon years of collaboration and private triumphs. It saddens me that you can only mention Moore's Law (which, while an acute observation by Moore, is not much of a tool) and large language models out of all the steps that we've already taken - and are still taking - towards furthering our understanding of our world.
And, thirdly and finally, what the fuck is up with this modern belief in AI singularity? Why do we believe that intelligence could ever be a runaway effect? That it is even plausible that anything could just recursively learn ad infinitum without limit? Have we all gotten so comfortable with the supposed singularities in black holes, that we've forgotten that singularities are a logical problem? That when the result of any postulate, any calculation, any thought, is a singularity, then one should probably reconsider? I am unconvinced that any form of intelligence singularity is likely or desireable. It seems to me too clean, too simple. It reeks of deification, of over-anthropomorphication. it sounds like the screams of sacrifices to volcano gods that could have gone on to study plate techtonics.
What is the point of this "singularity", what do we hope to gain? Why in all nine hells do we doggedly pursue it, seemingly without question?
My mind? Boggled.
Marketing - that's why. It's a central tent pole in the never-ending tech bro scam cycle. They are selling Roko's basilisk.
As for my own intuition on the matter... I tend to toss out the math and look at hardware. We know for a fact that consciousness, whatever it is, however it works, comes to an instant stop the moment you shut down the micro-tubules using anesthesia. We also know for a fact that this works just as well in a single cell organism as it does in any other living thing. There are no living things without these micro-tubules as part of their structure, it's in every cell. That means evolution itself has selected for them so strongly that we have no examples of life or consciousness or sapience without them.
There are no micro-tubules present in a computer. Further there is also no stand-in present for the quantum effects these micro-tubules generate, which is provably required to be present for consciousness. By this reasoning, there are no conscious machines, yet - and we wouldn't even expect to see any become possible until we are running light-based hardware that is designed to perform the same functions we see in living things.
Evolution has had several billion years of uninterrupted uptime to perform field testing more brutal than anything humans have ever done with code. I think it's a bit premature to assume that evolution itself hasn't already begun to approach some hardware optimization limits that machines wouldn't be able to push past with impunity. Nature may already be nearing those limits. Something better might be possible but we're sure not going to figure it out until after we answer the how and why of biological consciousness.
We've invented syncopathic librarians that are more convenient than search engines in some circumstances, and are suitable as a pocket grad-student in others. Too bad people still need to check all of their work all of the time. It might be enough to make robots about as useful as a teenager at doing the chores. If you want more than that out of it, you've got to specialize the diet you feed to them during training, that's still 'narrow'. It's a very useful tool that can be embedded into just about every other tool... but that's it. Something else is still missing just to bring it up to our level.
The current tech bubble will continue, it seems, until they manage to stuff the entire corpus of human knowledge into the context window, or they run out of funding. Meanwhile, all the AI power you'll likely need in anything you're doing yourself will come free, bundled in your next graphics card upgrade. I'd give it a couple years tops before someone figures out a way to train them better on those cards and on small (even tiny or micro) data sets. That's the point where people finally realize the data center model we're rushing now is a scam.
Strong words. In what sense are you using the word “proof” here?
In the sense that you can turn consciousness off and on like a light switch by doing nothing except interfering with whatever it is that happens inside of a micro-tubule. All other biology is unaffected and shutting down their activity stops consciousness. This works on humans and on single cell organisms, so this is a simpler problem at the root than brains are. The consciousness problem is locked in a box now, even though we don't understand how it works or why it works. The only logical explanation for this is for the activity in the micro-tubules to be a fundamental necessity for consciousness, and other factors like neuro-chemistry are already ruled out. Brains and all of the fancy things that neurons do were built on top, much later.
When we figure out what exactly is going on with this - a much easier task now that we can do it by studying single cell organisms - we'll be in a position to figure out what brains did to build up from this base and optimize it. Then we can begin to tie in neuro-chemistry and electrical activity and finally understand what is going on and why it gives rise to a conscious experience. That's when we'll be able to duplicate it in hardware, if we want to.
This reminds me of when people thought that flying could only ever be accomplished by birds and insects because only birds and insects could fly.
The fact is, we have relatively little idea what consciousness is or what its necessary preconditions are.
Yet we can turn it off and on with impunity in a way that shatters all prior theory with concrete scientific facts. That means we know where it is hiding in the biology for the first time. New information which completely redefines the problem space.
You can wave hands and quote platitudes about the impossibility of flight (which we could see happening with our own eyes even if we couldn't duplicate it) or about how people used to say there would never be color television. Everyone who studied these topics was better informed than the general public and smarter than base platitudes, which is why we have drones and 4KHD now.
Consciousness is just another problem to solve, the same way they were, and we've made real progress.
Turning consciousness off with anesthesia shows consciousness depends on certain brain functions continuing. It doesn’t tell you which substrate generates consciousness. Even if microtubules were affected, that would at most show they’re a necessary part of brain function - like electricity is necessary for a computer - but it wouldn’t prove microtubules are where consciousness “lives,” let alone that quantum microtubule effects are required.
Put another way, I can take a computer and stop it from working by removing any number of different parts. I could remove the wire connecting the power supply to the wall. I could remove the power supply. I could remove the CPU. I could remove the memory. I could remove the bus. I could remove the connection from the motherboard to the video card. I could remove the connection from the video card to the display. I could go and blow up the power grid. These would all have the same effect on my perception of whether the computer is working.
This is my point though, we can see consciousness happening too, and the way we made human flight happen was through a totally different mechanism than we saw with birds.
Also, and importantly, the mechanism birds use is not how we got to moving several times faster than the speed of sound. Likewise I suspect the mechanism we use for superhuman intelligence will be possibly biologically inspired, but not identical.
Turning a single cell organism like bacteria on and off with anesthesia has nothing to do with any brain function, because single cell critters haven't got brains to begin with. They do have memory, and react to their environment, so how are they handling that processing without a brain, and why does anesthesia shut that down? That's the point people keep missing in this research - brains have nothing to do with consciousness. They aren't required for it and we've proven that. The fundamentals fit into a single cell.
Bacteria are at the bottom of the curve, we're near the top on this planet but hardly alone in being able to recognize our reflections in a mirror. Animals have internal models of themselves and LLMs do not. I could list many aspects of consciousness they haven't got yet and will never get from matrix multiplication. We'd have to build those in as separate systems on top of the LLM just to catch them up to us. In fact we probably will.
I do think we'll get there, but not on this hardware, not a snowball's chance in hell. It'd be like building yourself an any-terrain vehicle and finding out it also doubles as a functional spaceship by sheer luck, turns out you just needed a bigger gas tank and some new tires to get to orbit. That's about as rational as expecting our current processors to handle an AGI workload.
I think LLMs tell us more about language and probability than consciousness. Everything they create is a mimic of the training data, the insight they provide when they provide it is because they hit on things we overlook. We're bad at going through terabytes of information rapidly and matching up small details in the training data, and they'll get better at it. That doesn't mean it's thinking - if it is, so is your pocket calculator, because all the LLM ever did to give you that answer was pick the largest numbers out of a matrix and they have the same hardware, just more of it.
We're trying to make it to AGI on hardware whose design hasn't changed appreciably in almost seventy years, hardware that does two things only - add numbers and execute a couple of basic logic operations. Repeating 'context window' like a mantra doesn't make an LLM sapient, though it's great for driving up memory prices once you've invested in companies that make chips. Occam's razor doesn't favor the odds of this waking up in some kind of omega moment. That's magical thinking.
The problem we've always had with the processor is answering the question, 'well, what do we replace the transistor with that can do a better job?' Nobody has ever had a good or even useful answer for that question. Basic logic and addition is a great place to start - but that doesn't mean nature bothers with either one of them, she could be doing something else we didn't think of yet. She uses physics to get to consciousness, while we are stuck using math to get there on computer hardware. That's a big disconnect.
I'm willing to bet that by the time we unwind the microtube mystery we're going to find nature's answer to the transistor/logic gate problem, and it'll take us in new, better design directions. We chase that and we may very well top human intelligence, since evolution optimizes for 'good enough' but rarely 'best' solutions. A single human mind at 8Hz on 20w of power can still out-think a hamlet-sized data center that's draining the lake and raising the town's electricity prices. This is not hardware to be proud of. It's got a terribly long way to go to catch up to us and it's taking up entirely too much energy.
That's where I'm at on all this, anyway.
The singularity has been postponed indefinitely, until better hardware arrives. ;)
As an aside, I know it's just a typo, but I love your new word.
I agree with @em-dash 's assessment with one other thing to add: a fundamental limitation of LLMs is context window. Once you reach this limit, the AI starts to forget things from earlier in the conversation.
Right now, the best I've seen is a million tokens, which is a lot, but isn't enough to hold larger code bases. There are some hacks, like breaking a problem down and dispatching the parts to sub-agents. But the context window places limits on how big a problem the LLM can take on.
Having worked with most of the high end models, I am confident that they are not effective for much without either significant guardrails or human oversight. By which I mean, somebody has to say, "yes that's what I wanted" or "no, it's wrong because" at the end of the process. Something like a test suite could replace that oversight, but then you've just moved the hard problem to writing bigger and more complex test suites.
I feel like there is a parallel here that I can't quite put into words. The idea that we shifted the work from synthesis to verification and the universe is reminding us there is no free lunch, there is a law of conservation somewhere here that seems to be happening. Something about NP!=P..
Essential complexity is the part of the problem that is intrinsically hard. This is in contrast to things you layer over the core problem. Think programming in assembly vs a high level language – you can solve the same programming problem in either, but one is easier to do than the other. But if the problem is the traveling salesman problem, no amount of high level language is going to fix the difficulty of that challenge.
In this context, the essential complexity is 1) knowing (precisely) what we want and 2) being able to express our want clearly and unambiguously to the machine.
Anthropic employees use Claude to code Claude: https://www.anthropic.com/research/how-ai-is-transforming-work-at-anthropic. The study indicates that it's increasing code output, but I'd be very skeptical, not only because it's from Anthropic but because the data is self-reported. More credible studies have demonstrated developers being less productive with AI agents like Claude Code, but I'm sure it depends on the person. Anecdotally, I mainly use Claude Code to write boilerplate, and I'm fairly confident it saves me more time than I spend writing the prompts and extra time debugging, because the boilerplate is easy to verify.
This means we may be getting to the singularity faster, but the curve isn't steep at least yet. You can't ask an LLM to improve itself 1000 times to get a super-LLM: there may be 1000 minor fixes and niche optimizations, but we almost definitely need at least one more breakthrough to get from a top-tier model (e.g. Opus 4.5) to human-level AGI, and current LLMs aren't discovering breakthroughs.
Well, actually...in a sort-of breakthrough, Erdos problem #728 was solved "more or less" fully autonomously by AI using a novel technique. The significance is discussed in the linked thread, most of which goes over my head, but my understanding is the AI still needed human assistance. More generally, cutting-edge researchers like Terrace Tao are using LLMs and reporting benefits, but (like with code) I haven't seen an explosion in discoveries, because the LLMs still need human guidance.
And something you may be interested in: what would happen if you asked Claude Code to improve a codebase 200 times? Someone did that and wrote about it. It's a short blog post, so rather than restating I recommend you read it. The tl;dr and relevance here is described well by its last sentence: "oh and the app still works, there's no new features, and just a few new bugs"
Honestly, I think this also overlooks that improving the actual code in terms of fixes or optimizations is not necessarily improving the LLM, definitely not in the sense implies by the whole singularity thing. The LLM isn't the Python code, it's the model. You can make myriad improvements to the app without ever making a single improvement to the model, and for the singularity to be a thing, an AI model would need to be improving the actual model itself. Claude could rewrite its entire codebase without ever doing that.
Models like this are improved by changing their architecture or how you train them (or what data you're training them on, but this is generally not what the singularity people are thinking of). You can make the actual code for Claude better from a software engineering perspective without remotely changing the underlying architecture of the model, much less the training approach. While LLMs can write the Python code for implementing existing approaches and architectures, especially in small/simple situations without a large codebase to interact with, there's no evidence of them even making incremental improvements on actually coming up with novel approaches to architecture or training that lead to improved outcomes -- and if there were, we'd be hearing about it.
The Erdos problem thing is cool but does not actually bear that much resemblance at all to what an AI would have to do to actually meaningfully improve itself (and the linked github at the end of that thread gives a lot of caveats in regards to AI contributions to Erdos problems).
"more or less" is doing a fuck load of lifting in that headline. A very very smart math person played middleman between GPT and Aristotole. That is NOT a more or less solve for AI. That is a human providing the literal reasoning and translation they cannot naturally do.
TLDR; I don’t expect a singularity any time soon.
There are two things to remember when you hear about the current and future capabilities of AI:
I’ve been using Claude code and copilot a lot recently and everything I’ve done follows this pattern:
There’s a good comic circulating about AI.
In the first panel, the person asks “is this mushroom edible?” And the AI answers “yes”.
In the second panel, the person is very ill and lying in a hospital bed. And the AI says “You're right, it's a poisonous mushroom. Would you like to learn more about poisonous mushrooms?”
The creator of the lovely Pepper and Carrot webcomics has recently been posting a fun series of mini-four panel comics on his blog mocking AI with an unnamed "gothic sorceress" and the "Avian Intelligence" her magic university has assigned her (in the first of the series).
One of them involves her blowing herself up with a
fireballself-immolation spell it gives her. Another has it suggesting a poison.I'm pretty confident that LLMs have already plateaued and will see little substantial improvement in the next 10 years. And by substantial, I specifically mean "things that feel like sci-fi". They'll become more useful and more efficient within the same paradigm, but there won't be any earth-shattering developments.
If AGI ever happens, I am not convinced it'll have much to do with LLMs. LLMs are fancy probability engines. They are to me closer to the Mechanical Turk than to true intelligence.
That said, I'm a science fiction guy who reads about some of that stuff. Not a STEM person.
I would say the same about people.
The Mechanical Turk was literally true intelligence though? I’m not sure you are thinking of the right analogy.
The mechanical Turk was a chess playing “computer” that had an actual human hiding inside it generating the moves.
My prediction is that LLMs will ultimately be a significant part of true AGI but the secret sauce is going to be a novel formulation of reinforcement learning.
It's the ELIZA Effect. Weizenbaum even described it as "disturbing," how people he otherwise respected would attribute intelligence to something he knew full well just repeated back what was said to it.
As others noted, LLMs don't exactly have "code". However, LLMs have been used as a part of the LLM training process for some time. In that sense, yes absolutely.
In another sense: the agent harness, which is an increasingly important part of how effective LLM powered agents are at real world tasks, also yes. The big model companies use coding agents extensively when creating the harness and scaffolding, which shows in the sheer volume of bugs the harnesses have.
I didn't read the article you linked so I may not have all the nuance about what intelligence explosion refers to. But in terms of practical application of AI agents at real world tasks we are undoubtedly in the midst of an intelligence explosion.
In terms of actual intelligence, as we might define it, LLMs arguably have none at all. It depends on how you frame it. They provide an illusion of intelligence that is so good they can do intelligent things, which is to say things that could previously only be accomplished with human intelligence. But it's achieved, essentially, through advanced pattern matching rather than anything that could be described as understanding. It's hard to imagine a path from there to true AGI but at the same time it's not difficult to imagine that at some point the illusion of intelligence could get so good that it's practically the same as the real thing for many applications.
It's all so weird.
The article is a side-note. The "intelligence explosion" is the idea that an AI is, first of all, unarguably "intelligent", and in fact intelligent enough to be able to improve it's own code base better/faster than a human could.
We are definitely not there yet, and in fact, I still don't think LLMs will ever get there, not w/o at least a couple more paradigm-shifting breakthroughs by humans, first.
But I am beginning to think that, indirectly, the LLMs may be to the point of helping humans reach those breakthroughs faster than we would, w/o them.
Ah, yes in that context we're nowhere near an explosion. Or at least the existing technology doesn't put us near one, who knows if there will be breakthroughs in the near future.
Yes LLMs are already helping move the technology along faster than humans alone could do it. I don't think there's any doubt of that. The only question is if the path leads to the vicinity of AGI, which I think is safe to answer yes. It doesn't matter of LLMs themselves will have anything to do with AGI, they will definitely accelerate many aspects of technological advancement and some of them will contribute to eventual AGI.
Yes, it's naturally the first thing you'd do.
To be clear we need to be very specific about what improvement means. The goal is to get a better LLM output.
I could take a LLM source code right now, find and replace every single variable with a dumb spelling mistake, and that code could run over itself and say "hey maybe don't spell things wrong" and make "better code" but it is NOT a better output.
In fact, we already have code that does that...called a spell checker. We've also got a compile time checking, linters, and intellisense. All "AI" by some definitions and "tools" by others. These can all be used to make better versions of themselves, and even with better output sometimes, but not without new input.
And that's basically where the LLM fails (well one of many spots). An LLM in a closed loop CANNOT produce better code. You can feed it more information and it could maybe produce better code, but that code will be code someone else developed first and you fed into the model.
It helps to remember that LLMs are very very very fancy autocomplete. They do not reason. They do not create. They just have a VAST pile of "reference material" that boils down to "well one time someone else was here and they did this".
It can maybe tease out small optimizations that technically existed but no one had put together, but it can't go beyond that. It requires someone else to make any sort of real discovery first for it to attempt to figure out and apply. This is very very very obvious when you're using a less popular language and when you look at the behaviors of these AIs in other venues.
One of the most interesting examples was OpenAI's DotA2 runs. Despite being HEAVILY marketed to only allow good news out, it's worth mentioning they had to hard code certain behaviors (creep blocking) and it never learned things like playing the eco game. These models have massive blindspots because again, they do not reason. I know there's articles here and there on hackernews and the like claiming stuff, but its all so full of bias and excuses.
I would bet far more money then I have that LLMs are truly incapable of anything close to AGI let alone a super-intelligent god. Firstly, they need more memory and more layers of memory. Models are frozen windows into their training data; I can't imagine a General Intelligence that is incapable of adapting its world-view with new data. I'm also knowledgable enough about LLMs to be pretty certain its impossible for them to do so. Secondly, real-world intelligence is far messier then stories about AGI generally engage with and way more then LLM companies engage with. Thirdly, I suspect that hardware is a hard blocking problem for reaching anything close to AGI, and no, I don't think a trillion dollars of datacenters is going to solve that problem.