This was kind of a hard read for me. I love Ted Chiang and think he's goated as an author but I just can't find myself aligning with this argument at all. Particularly the idea that a sufficiently...
This was kind of a hard read for me. I love Ted Chiang and think he's goated as an author but I just can't find myself aligning with this argument at all. Particularly the idea that a sufficiently smart person can't make someone around them smarter is just not even a little relevant. A closer parallel would be if a person could somehow take a large set of clones of themselves, modify each ones genome slightly, and then kill off all but the smartest variation. Rinse, repeat ad infinitum. As far as I can tell there's nothing preventing this from working in theory. This is exactly what an AI would be capable of doing and it doesn't even need to fuss about with pesky biological processes.
Yes, his argument is too biologically grounded. Modifying "software"* is much easier and we are able to do in much faster iterations than modifying "wetware". The results are also much more...
Yes, his argument is too biologically grounded. Modifying "software"* is much easier and we are able to do in much faster iterations than modifying "wetware". The results are also much more observable and testable, not mentioning the moral aspect of experimenting on humans.
artificial neural networks, which are the current state of the art implementation of the "AI" are not entirely software in a classic sense, but the rest still holds.
What baffles me most about his take is that he wrote The Lifecycle of Software Objects, where they do exactly what you just described to generate their a-life!
What baffles me most about his take is that he wrote The Lifecycle of Software Objects, where they do exactly what you just described to generate their a-life!
This argument seems a bit surprising coming from the author of "Understand," but I suppose it's fair to say that being able to imagine superintelligence isn't enough to justify believing in it. I...
This argument seems a bit surprising coming from the author of "Understand," but I suppose it's fair to say that being able to imagine superintelligence isn't enough to justify believing in it. I wonder if he's changed his opinion any since 2021?
Many of his stories are classic examples of "what if" - taking place in internally consistent universes which however behave notably differently from our universe in a selected aspect. "What if...
Many of his stories are classic examples of "what if" - taking place in internally consistent universes which however behave notably differently from our universe in a selected aspect. "What if religion didn't require faith?" "What if the way we develop and use language influenced the way we perceive time?" "What if human intelligence explosion was possible?"
I think that is one of my favorite stories of his. What is surprising to you about its connection to this article? It has been a while since I have read it, but I don’t remember it being written...
I think that is one of my favorite stories of his. What is surprising to you about its connection to this article? It has been a while since I have read it, but I don’t remember it being written in a way that suggests how ultraintelligence would be a real possibility. I’ll have to give it a re-read!
In "understand" (great short story btw if anyone is interested) the protagonist gets an experimental drug that make him significantly smarter, he uses his intelligence to somehow steal more of the...
In "understand" (great short story btw if anyone is interested) the protagonist gets an experimental drug that make him significantly smarter, he uses his intelligence to somehow steal more of the drug to get even smarter, then reinvents a new language to think more clearly in, then thinks about building a computer to overcome the limitation of a human brain.
I mean, it's just fiction, but I truly assumed he was talking about recursively self improving AI when I read this story.
Ah okay, I see what you mean. That definitely makes sense! I'll put the rest of my response here to avoid spoiling the story for anyone I think when I read it some years back I read it as a...
Ah okay, I see what you mean. That definitely makes sense!
I'll put the rest of my response here to avoid spoiling the story for anyone
I think when I read it some years back I read it as a metaphor for what "knowledge is power" means in the 21st century; whether it is AI, Google & Friends slurping up everyone's "behavioral surplus," or some dude jacked up on smart pills :b The plot revolves around how the protagonist's increased intelligence/knowledge suddenly necessitates their having to immediately spend all their energy trying to gain more knowledge and intelligence as a matter of self-preservation. I can't remember the exact details, but I vaguely remember that the other super-intelligent character is described by the narrator as being less virtuous or righteous because of their wanting to use their powers for good (or something like that) as if the protagonist is trying to not only defend themself but humanity as a whole. But they are both effectively the same kind of hazard to society.
Another thing that stood out to me is the narrator's first-person descriptions of becoming super-intelligent. Maybe because my field is the philosophy of education, I read the story as kind of a speculative commentary on the nature of intelligence and maybe experience more generally. I recall him describing his higher intelligence as basically alogical, describing his learning process as a kind of aesthetic experience. Having not read the story in forever, these sorts of details are what remain most vividly in my memory and apparently made a big impression on me at the time. I'll have to give it another go, if only to enjoy the build up to that awesome ending :)
I appreciate Ted Chiang’s thoughts and think this was a solid argument against the intelligence explosion… 5 years ago. Now, it reads more like a prediction piece describing how impossibly hard...
I appreciate Ted Chiang’s thoughts and think this was a solid argument against the intelligence explosion… 5 years ago. Now, it reads more like a prediction piece describing how impossibly hard the things LLMs are doing right now is.
For example, there are plenty of people who have I.Q.s of 130, and there’s a smaller number of people who have I.Q.s of 160. None of them have been able to increase the intelligence of someone with an I.Q. of 70 to 100, which is implied to be an easier task.
A group of A.I. researchers created something far, far smarter than any one of them individually. (Even if the nature of intelligence in LLMs is dubious, Ted Chiang is making no claims about the nature of intelligence, only the observables. Modern LLMs, despite their distinctly inhuman flaws like hallucinations, are generally far smarter in observation than most humans.)
Some proponents of an intelligence explosion argue that it’s possible to increase a system’s intelligence without fully understanding how the system works. They imply that intelligent systems, such as the human brain or an A.I. program, have one or more hidden “intelligence knobs,” and that we only need to be smart enough to find the knobs. I’m not sure that we currently have many good candidates for these knobs, so it’s hard to evaluate the reasonableness of this idea.
Whoops! Compute-scaling.
An individual working in complete isolation can come up with a breakthrough but is unlikely to do so repeatedly; you’re better off having a lot of people drawing inspiration from one another. They don’t have to be directly collaborating; any field of research will simply do better when it has many people working in it.
Whoops! AI programs now use many agents to get the “random search” effect, all searching differently for the same good idea.
A few A.I. programs have been designed to play a handful of similar games, but the expected range of inputs and outputs is still extremely narrow. Now, alternatively, suppose that you’re writing an A.I. program and you have no advance knowledge of what type of inputs it can expect or of what form a correct response will take. In that situation, it’s hard to optimize performance, because you have no idea what you’re optimizing for. How much can you optimize for generality? To what extent can you simultaneously optimize a system for every possible situation, including situations never encountered before?
Whoops! Multi-modal models.
I don’t think Ted Chiang is somehow shortsighted for this piece. I simply think he wasn’t working at DeepMind or OpenAI or a university AI lab when he wrote this piece. This is a great example of why experts are often worth listening to. Fantastic, well-articulated arguments backed by hundreds of years of history can be so, so wrong when one does not know where innovation is headed.
More compute doesn't necessarily translate into better results, however. Anyone who's ever trained a neutral network knows that the loss tends to level off eventually, and that if you want to...
Whoops! Compute-scaling.
More compute doesn't necessarily translate into better results, however. Anyone who's ever trained a neutral network knows that the loss tends to level off eventually, and that if you want to improve performance, you either have to retune your hyperparameters or obtain better data. But tuning hyperparameters can only take you so far, and there is only so much quality data that exists. That implies an upper limit somewhere.
Perhaps LLMs, having been trained on human text, will plateau under optimized circumstances to the theoretical upper limit for human intelligence. That would be incredibly useful, to be sure, but definitionally not superintelligence.
Or to be even more speculative, perhaps human-level intelligence is near the theoretical upper limit of biological or even "computational" intelligence. (One can always wonder why humans didn't evolve to be cleverer.) Under this regime, superintelligence would be impossible to achieve regardless of technique or technology.
For now we're squarely in the non-superintelligence era. LLMs can augment human work, turning the output of a non-expert to something more advanced, but it doesn't produce results that are unachievable by human cognition. The LLM responses might come faster, but for the same reason that you can't "run" a rat faster and have it produce quantum mechanics, the near-instantaneous responses of an LLM don't prove that LLMs are more capable. We won't be able to conclude that superintelligence exists until LLMs (or something else) can produce objectively correct results whose derivations are inscrutable to domain experts.
This piece was written 4 months after the release of chat gpt so in many ways is a response to modern LLM AI. By then the writing was on the wall that these things would be a huge deal and...
this was a solid argument against the intelligence explosion… 5 years ago. Now, it reads more like a prediction piece describing how impossibly hard the things LLMs are doing right now is.
This piece was written 4 months after the release of chat gpt so in many ways is a response to modern LLM AI. By then the writing was on the wall that these things would be a huge deal and growth/investment was inevitable so I give him less argumentative leeway here.
Regardless, the fact that we're often only a technological or mindset shift away from the once impossible now within grasp I find these "it'll never happen" pieces pointless at best and damaging at worst when the potential risk on the other end of the spectrum is "If Anyone Builds It, Everyone Dies"
I think the article does work in the sense of “it’s not proven and imagining it isn’t the same as proving it.” But such an argument doesn’t provide any constraints on what might happen, and we’re...
I think the article does work in the sense of “it’s not proven and imagining it isn’t the same as proving it.” But such an argument doesn’t provide any constraints on what might happen, and we’re unlikely to get any. There’s no law of physics we can use to put bounds on what AI might do someday.
What might recursive self-improvement look like for human beings? For the sake of convenience, we’ll describe human intelligence in terms of I.Q., not as an endorsement of I.Q. testing but because I.Q. represents the idea that intelligence can be usefully captured by a single number, this idea being one of the assumptions made by proponents of an intelligence explosion. In that case, recursive self-improvement would look like this: Once there’s a person with an I.Q. of, say, 300, one of the problems this person can solve is how to convert a person with an I.Q. of 300 into a person with an I.Q. of 350. And then a person with an I.Q. of 350 will be able to solve the more difficult problem of converting a person with an I.Q. of 350 into a person with an I.Q. of 400. And so forth.
Do we have any reason to think that this is the way intelligence works? I don’t believe that we do. For example, there are plenty of people who have I.Q.s of 130, and there’s a smaller number of people who have I.Q.s of 160. None of them have been able to increase the intelligence of someone with an I.Q. of 70 to 100, which is implied to be an easier task. None of them can even increase the intelligence of animals, whose intelligence is considered to be too low to be measured by I.Q. tests. If increasing someone’s I.Q. were an activity like solving a set of math puzzles, we ought to see successful examples of it at the low end, where the problems are easier to solve. But we don’t see strong evidence of that happening.
[...]
Obviously, none of this proves that an intelligence explosion is impossible. Indeed, I doubt that one could prove such a thing, because such matters probably aren’t within the domain of mathematical proof. This isn’t a question of proving that something is impossible; it’s a question of what constitutes good justification for belief. The critics of Anselm’s ontological argument aren’t trying to prove that there is no God; they’re just saying that Anselm’s argument doesn’t constitute a good reason to believe that God exists. Similarly, a definition of an “ultraintelligent machine” is not sufficient reason to think that we can construct such a device.
[...]
The rate of innovation is increasing and will continue to do so even without any machine able to design its successor. Some might call this phenomenon an intelligence explosion, but I think it’s more accurate to call it a technological explosion that includes cognitive technologies along with physical ones. Computer hardware and software are the latest cognitive technologies, and they are powerful aids to innovation, but they can’t generate a technological explosion by themselves. You need people to do that, and the more the better. Giving better hardware and software to one smart individual is helpful, but the real benefits come when everyone has them. Our current technological explosion is a result of billions of people using those cognitive tools.
This was kind of a hard read for me. I love Ted Chiang and think he's goated as an author but I just can't find myself aligning with this argument at all. Particularly the idea that a sufficiently smart person can't make someone around them smarter is just not even a little relevant. A closer parallel would be if a person could somehow take a large set of clones of themselves, modify each ones genome slightly, and then kill off all but the smartest variation. Rinse, repeat ad infinitum. As far as I can tell there's nothing preventing this from working in theory. This is exactly what an AI would be capable of doing and it doesn't even need to fuss about with pesky biological processes.
Yes, his argument is too biologically grounded. Modifying "software"* is much easier and we are able to do in much faster iterations than modifying "wetware". The results are also much more observable and testable, not mentioning the moral aspect of experimenting on humans.
What baffles me most about his take is that he wrote The Lifecycle of Software Objects, where they do exactly what you just described to generate their a-life!
This argument seems a bit surprising coming from the author of "Understand," but I suppose it's fair to say that being able to imagine superintelligence isn't enough to justify believing in it. I wonder if he's changed his opinion any since 2021?
Many of his stories are classic examples of "what if" - taking place in internally consistent universes which however behave notably differently from our universe in a selected aspect. "What if religion didn't require faith?" "What if the way we develop and use language influenced the way we perceive time?" "What if human intelligence explosion was possible?"
I think that is one of my favorite stories of his. What is surprising to you about its connection to this article? It has been a while since I have read it, but I don’t remember it being written in a way that suggests how ultraintelligence would be a real possibility. I’ll have to give it a re-read!
In "understand" (great short story btw if anyone is interested) the protagonist gets an experimental drug that make him significantly smarter, he uses his intelligence to somehow steal more of the drug to get even smarter, then reinvents a new language to think more clearly in, then thinks about building a computer to overcome the limitation of a human brain.
I mean, it's just fiction, but I truly assumed he was talking about recursively self improving AI when I read this story.
Ah okay, I see what you mean. That definitely makes sense!
I'll put the rest of my response here to avoid spoiling the story for anyone
I think when I read it some years back I read it as a metaphor for what "knowledge is power" means in the 21st century; whether it is AI, Google & Friends slurping up everyone's "behavioral surplus," or some dude jacked up on smart pills :b The plot revolves around how the protagonist's increased intelligence/knowledge suddenly necessitates their having to immediately spend all their energy trying to gain more knowledge and intelligence as a matter of self-preservation. I can't remember the exact details, but I vaguely remember that the other super-intelligent character is described by the narrator as being less virtuous or righteous because of their wanting to use their powers for good (or something like that) as if the protagonist is trying to not only defend themself but humanity as a whole. But they are both effectively the same kind of hazard to society.
Another thing that stood out to me is the narrator's first-person descriptions of becoming super-intelligent. Maybe because my field is the philosophy of education, I read the story as kind of a speculative commentary on the nature of intelligence and maybe experience more generally. I recall him describing his higher intelligence as basically alogical, describing his learning process as a kind of aesthetic experience. Having not read the story in forever, these sorts of details are what remain most vividly in my memory and apparently made a big impression on me at the time. I'll have to give it another go, if only to enjoy the build up to that awesome ending :)
I appreciate Ted Chiang’s thoughts and think this was a solid argument against the intelligence explosion… 5 years ago. Now, it reads more like a prediction piece describing how impossibly hard the things LLMs are doing right now is.
A group of A.I. researchers created something far, far smarter than any one of them individually. (Even if the nature of intelligence in LLMs is dubious, Ted Chiang is making no claims about the nature of intelligence, only the observables. Modern LLMs, despite their distinctly inhuman flaws like hallucinations, are generally far smarter in observation than most humans.)
Whoops! Compute-scaling.
Whoops! AI programs now use many agents to get the “random search” effect, all searching differently for the same good idea.
Whoops! Multi-modal models.
I don’t think Ted Chiang is somehow shortsighted for this piece. I simply think he wasn’t working at DeepMind or OpenAI or a university AI lab when he wrote this piece. This is a great example of why experts are often worth listening to. Fantastic, well-articulated arguments backed by hundreds of years of history can be so, so wrong when one does not know where innovation is headed.
More compute doesn't necessarily translate into better results, however. Anyone who's ever trained a neutral network knows that the loss tends to level off eventually, and that if you want to improve performance, you either have to retune your hyperparameters or obtain better data. But tuning hyperparameters can only take you so far, and there is only so much quality data that exists. That implies an upper limit somewhere.
Perhaps LLMs, having been trained on human text, will plateau under optimized circumstances to the theoretical upper limit for human intelligence. That would be incredibly useful, to be sure, but definitionally not superintelligence.
Or to be even more speculative, perhaps human-level intelligence is near the theoretical upper limit of biological or even "computational" intelligence. (One can always wonder why humans didn't evolve to be cleverer.) Under this regime, superintelligence would be impossible to achieve regardless of technique or technology.
For now we're squarely in the non-superintelligence era. LLMs can augment human work, turning the output of a non-expert to something more advanced, but it doesn't produce results that are unachievable by human cognition. The LLM responses might come faster, but for the same reason that you can't "run" a rat faster and have it produce quantum mechanics, the near-instantaneous responses of an LLM don't prove that LLMs are more capable. We won't be able to conclude that superintelligence exists until LLMs (or something else) can produce objectively correct results whose derivations are inscrutable to domain experts.
This piece was written 4 months after the release of chat gpt so in many ways is a response to modern LLM AI. By then the writing was on the wall that these things would be a huge deal and growth/investment was inevitable so I give him less argumentative leeway here.
Regardless, the fact that we're often only a technological or mindset shift away from the once impossible now within grasp I find these "it'll never happen" pieces pointless at best and damaging at worst when the potential risk on the other end of the spectrum is "If Anyone Builds It, Everyone Dies"
I think the article does work in the sense of “it’s not proven and imagining it isn’t the same as proving it.” But such an argument doesn’t provide any constraints on what might happen, and we’re unlikely to get any. There’s no law of physics we can use to put bounds on what AI might do someday.
https://archive.is/BP606
From the article:
[...]
[...]