What might recursive self-improvement look like for human beings? For the sake of convenience, we’ll describe human intelligence in terms of I.Q., not as an endorsement of I.Q. testing but because I.Q. represents the idea that intelligence can be usefully captured by a single number, this idea being one of the assumptions made by proponents of an intelligence explosion. In that case, recursive self-improvement would look like this: Once there’s a person with an I.Q. of, say, 300, one of the problems this person can solve is how to convert a person with an I.Q. of 300 into a person with an I.Q. of 350. And then a person with an I.Q. of 350 will be able to solve the more difficult problem of converting a person with an I.Q. of 350 into a person with an I.Q. of 400. And so forth.
Do we have any reason to think that this is the way intelligence works? I don’t believe that we do. For example, there are plenty of people who have I.Q.s of 130, and there’s a smaller number of people who have I.Q.s of 160. None of them have been able to increase the intelligence of someone with an I.Q. of 70 to 100, which is implied to be an easier task. None of them can even increase the intelligence of animals, whose intelligence is considered to be too low to be measured by I.Q. tests. If increasing someone’s I.Q. were an activity like solving a set of math puzzles, we ought to see successful examples of it at the low end, where the problems are easier to solve. But we don’t see strong evidence of that happening.
[...]
Obviously, none of this proves that an intelligence explosion is impossible. Indeed, I doubt that one could prove such a thing, because such matters probably aren’t within the domain of mathematical proof. This isn’t a question of proving that something is impossible; it’s a question of what constitutes good justification for belief. The critics of Anselm’s ontological argument aren’t trying to prove that there is no God; they’re just saying that Anselm’s argument doesn’t constitute a good reason to believe that God exists. Similarly, a definition of an “ultraintelligent machine” is not sufficient reason to think that we can construct such a device.
[...]
The rate of innovation is increasing and will continue to do so even without any machine able to design its successor. Some might call this phenomenon an intelligence explosion, but I think it’s more accurate to call it a technological explosion that includes cognitive technologies along with physical ones. Computer hardware and software are the latest cognitive technologies, and they are powerful aids to innovation, but they can’t generate a technological explosion by themselves. You need people to do that, and the more the better. Giving better hardware and software to one smart individual is helpful, but the real benefits come when everyone has them. Our current technological explosion is a result of billions of people using those cognitive tools.
This argument seems a bit surprising coming from the author of "Understand," but I suppose it's fair to say that being able to imagine superintelligence isn't enough to justify believing in it. I...
This argument seems a bit surprising coming from the author of "Understand," but I suppose it's fair to say that being able to imagine superintelligence isn't enough to justify believing in it. I wonder if he's changed his opinion any since 2021?
https://archive.is/BP606
From the article:
[...]
[...]
This argument seems a bit surprising coming from the author of "Understand," but I suppose it's fair to say that being able to imagine superintelligence isn't enough to justify believing in it. I wonder if he's changed his opinion any since 2021?