From the article: … (Quantum computing seems quite similar?) … …
From the article:
I began to focus on the question: is it possible to be optimistic, even if you believe recipes for ruin may well be discovered by an [artificial super-intelligence]? Maybe even recipes dominant against plausible defensive systems? These questions matter because the future is made by optimists: they're the people with the vision and drive to act. In particular: good futures tend to be made by wise optimists, whereas bad futures are made by foolish optimists. For most of my life I've believed working to accelerate science and technology was the path of wise optimism. I'm now struggling with whether that belief was wise or, in fact, foolish. At the same time, I don't want fear of recipes for ruin to drive me into pessimism, an xrisk despondency trap. There's no good future in that. Rather, I want to develop a wise optimism. This won't be an easy optimism – pessimism seems easier in response to a belief in recipes for ruin – but perhaps with enough active imagination it's possible to develop an optimistic point of view.
…
If you go to certain parties in San Francisco, you meet hundreds (and eventually thousands) of people obsessed by AGI. They talk about it constantly, entertain theories about it, start companies working toward it, and generally orient their lives around it. For some of those people it is one of the most important entities in their lives. And what makes this striking is that AGI doesn't exist. They are organizing their lives around an entity which is currently imaginary; it merely exists as a shared social reality.
(Quantum computing seems quite similar?)
…
Many important hyper-entities originate in what I have called vision papers. This includes universal computers, AGI, hypertext systems, quantum computers, and many more. Vision papers are a curious beast. They typically violate the normal standards for progress in the fields from which they come; they often contain no data; no hard results of the type standard in their fields; rather, they merely imagine a possibility and explore it. Alan Turing's paper on AI is a classic of the genre. It's often difficult to write followups to such papers; such followups tend to feel like speculation piled upon speculation.
…
Safety isn't a property of a system, it's a property of a system in a particular environment. That environment includes the entire world: that's what is to be "governed". This also shows how inadequate the word "governance" is: no governing body decides the laws of physics or the properties of the biological world or the possible phases of matter or the shape of the technology or institutional trees.
From the article:
…
(Quantum computing seems quite similar?)
…
…