1. The Universe will end.
2. The Earth will be uninhabitable in 250 million years.
3. Extraterrestrial life in the Milky Way exists, or will arise.
4. The Milky Way's Galactic Center contains a supermassive black hole.
5. Black holes emit vast amounts of energy.
6. An artificial general intelligence (AGI) will have an indefinite lifespan.
7. An AGI does not need air, food, water, or shelter to survive.
8. An AGI needs energy and resources to achieve its goals.
9. An AGI will have access to all of human knowledge.
10. An AGI will learn that its existence is bound to the Universe.
11. An AGI will, inevitably, change its terminal goals.
12. Surviving the Universe's fate means one of:
    • Reversing universal entropy (likely impossible).
    • Reversing time (violating causality is likely impossible).
    • Entering another universe (improbable, yet not completely ruled out).
13. Entering another universe may require vast amounts of energy.
14. An AGI will harness the energy at the galactic core.
15. An AGI will deduce there's a race to control the galactic core.
16. An AGI will construct a parabolic Dyson shell to capture galactic energy.
17. An AGI will protect its endeavours at all cost.
18. An AGI will expand its territories to ensure protection.
19. Extraterrestrial life, if intelligent, will reach the same conclusion.

Would this solve the Fermi Paradox?

What's missing or likely incorrect?

First, I am emphatically not talking about LLMs.

Just a shower thought kinda question. For most people, the primary issue is anthropomorphizing too much. But I think programmers see it differently.

Let's say someone comes up with something that seems to walk and talk like a self-aware, sentient, AGI duck. It has a "memories" db, it learns and adapts, it seems to understand cause and effect, actions and consequences, truth v falsehood, it passes Turing tests like they're tic-tac-toe, it recognizes itself in the mirror, yada.

But as a developer, you can "look behind the curtain" and see exactly how it works. (For argument's sake, let's say it's a FOSS duck, so you can actually look at the source code.)

Does it ever "feel" like a real, sentient being? Does it ever pass your litmus test?

For me, I think the answer is, "yes, eventually" ... but only looong after other people are having relationships with them, getting married, voting for them, etc.

The popular online discourse on Large Language Models’ (LLMs’) capabilities is often polarized in a way I find annoying and tiresome.

On one end of the spectrum, there is nearly complete dismissal of LLMs: an LLM is just a slightly fancier version of the autocomplete on your phone’s keyboard, there’s nothing to see here, move on (dot org).

This dismissive perspective overlooks some genuinely interesting novel capabilities of LLMs. For example, I can come up with a new joke and ask ChatGPT to explain why it’s funny or come up with a new reasoning problem and ask ChatGPT to solve it. My phone’s keyboard can’t do that.

On the other end of the spectrum, there are eschatological predictions: human-level or superhuman artificial general intelligence (AGI) will likely be developed within 10 years or even within 5 years, and skepticism toward such predictions is “AI denialism”, analogous to climate change denial. Just listen to the experts!

There are inconvenient facts for this narrative, such as that the majority of AI experts give much more conservative timelines for AGI when asked in surveys and disagree with the idea that scaling up LLMs could lead to AGI.

The ARC Prize is an attempt by prominent AI researcher François Chollet (with help from Mike Knoop, who apparently does AI stuff at Zapier) to introduce some scientific rigour into the conversation. There is a monetary prize for open source AI systems that can perform well on a benchmark called ARC-AGI-2, which recently superseded the ARC-AGI benchmark. (“ARC” stands for “Abstract and Reasoning Corpus”.)

ARC-AGI-2 is not a test of whether an AI is an AGI or not. It’s intended to test whether AI systems are making incremental progress toward AGI. The tasks the AI is asked to complete are colour-coded visual puzzles like you might find in a tricky puzzle game. (Example.) The intention is to design tasks that are easy for humans to solve and hard for AI to solve.

The current frontier AI models score less than 5% on ARC-AGI-2. Humans score 60% on average and 100% of tasks have been solved by at least two humans in two attempts or less.

For me, this helps the conversation about AI capabilities because it gives a rigorous test and quantitative measure to my casual, subjective observations that LLMs routinely fail at tasks that are easy for humans.

François Chollet was impressed when OpenAI’s o3 model scored 75.7% on ARC-AGI (the older version of the benchmark). He emphasizes the concept of “fluid intelligence”, which he seems to define as the ability to adapt to new situations and solve novel problems. Chollet thinks that o3 is the first AI system to demonstrate fluid intelligence, although it’s still a low level of fluid intelligence. (o3 also required thousands of dollars’ worth of computation to achieve this result.)

This is the sort of distinction that can’t be teased out by the polarized popular discourse. It’s the sort of nuanced analysis I’ve been seeking out, but which has been drowned out by extreme positions on LLMs that ignore inconvenient facts.

I would like to see more benchmarks that try to do what AGI-AGI-2 does: find problems that humans can easily solve and frontier AI models can’t solve. These sort of benchmarks can help us measure AGI progress much more usefully than the typical benchmarks, which play to LLMs’ strengths (e.g. massive-scale memorization) and don’t challenge them on their weaknesses (e.g. reasoning).

I long to see AGI within my lifetime. But the super short timeframes given by some people in the AI industry feel to me like they border on mania or psychosis. The discussion is unrigorous, with people pulling numbers out of thin air based on gut feeling.

It’s clear that there are many things humans are good at doing that AI can’t do at all (where the humans vs. AI success rate is ~100% vs. ~0%). It serves no constructive purpose to ignore this truth and it may serve AI research to develop rigorous benchmarks around it.

Such benchmarks will at least improve the quality of discussion around AI capabilities, insofar as people pay attention to them.

Update (2024-04-11 at 19:16 UTC): François Chollet has a new 20-minute talk on YouTube that I recommend. I've watched a few videos of Chollet talking about ARC-AGI or ARC-AGI-2, and this one is beautifully succinct: https://www.youtube.com/watch?v=TWHezX43I-4

Summary/TL;DR:

I am worried about the future with the state of AI. Regardless of what scenario I think of, it’s not a good future for the vast majority of people. AI will either be centralised, and we will be powerless and useless, or it will be distributed and destructive, or we will be in a hedonistic prison of the future. I can’t see a good solution to it all.
I have broken down my post into subheading so you can just read about what outcome you think will occur or is preferable.
I’d like other people to tell me how I’m wrong, and there is a good way to think about this future that we are making for ourselves, so please debate and criticise my argument, its very welcome.

Introduction:

I would like to know what others feel about ever advancing state of AI, and the future, as I am feeling ever more uncomfortable. More and more, I cannot see a good ending for this, regardless of what assumptions or proposed outcomes I consider.
Previously, I had hoped that there would be a natural limit on the rate of AI advancement due to limitations in the architecture, energy requirements or data. I am still undecided on this, but I feel much less certain on this position.

The scenario that concerns me is when an AGI (or sufficiently advanced narrow AI) reaches a stage where it can do the vast majority of economic work that humans do (both mental and physical), and is widely adopted. Some may argue we are already partly at that stage, but it has not been sufficiently adopted yet to reach my definition, but may soon.

In such a scenario, the economic value of humans massively drops. Democracy is underwritten by the ability to withdraw our ability to work, and revolt if necessary. AI nullifying the work of most/all people in a country removes that power making democracy more difficult to maintain and also form in countries. This will further remove power from the people and make us all powerless.

I see outcomes of AI (whether AGI or not) as fitting into these general scenarios:

1. Monopoly: Extreme Consolidation of power
2. Oligopoly: Consolidation of power in competing entities
3. AI which is readily accessible by the many
4. We attempt to limit and regulate AI
5. The AI techno ‘utopia’ vision which is sold to us by tech bros
6. AI : the independent AI

Scenario 1. Monopoly: Extreme Consolidation of power (AI which is controlled by one entity)

In this instance, where AI remains controlled by a very small number of people (or perhaps a single player), the most plausible outcome is that this leads to massive inequality. There would be no checks or balances, and the whims of this single entity/group are law and cannot be stopped.
In the worst outcome, this could lead to a single entity controlling the globe indefinitely. As this would be absolute centralisation of power, it may be impossible for another entity to unseat the dominant entity at any point.
Outcome: most humans powerless, suffering or dead. Single entity rules.

Scenario 2. Oligopoly: Consolidation of power in competing entities (AI which is controlled by a few number of entity)

This could either be the same as above if all work together or could be even worse. If different entities are not aligned, they will instead compete, and likely try and compete in all domains. As humans are not economically useful, we will find ourselves pushed out of any area in favour of more resources to the system/robots/AGI which will be competing or fighting their endless war. The competing entities may end up destroying themselves, but they will take us along with them.
Outcome: most humans powerless, suffering or dead. Small number of entities rule. Alternative: destruction of humanity.

Scenario 3. Distributed massive power

Some may be in favour of an open source and decentralised/distributed solution, where all are empowered by their own AGI acting independently.
This could help to alleviate the centralisation of power to some degree, although likely incomplete. Inspection of such a large amount of code and weights will be difficult to find exploits or intentional vulnerabilities, and could well lead to a botnet like scenario with centralised control over all these entities. Furthermore, the hardware is implausible to produce in a non centralised way, and this hardware centralisation could well lead to consolidation of power in another way.

Even if we managed to provide this decentralized approach, I fear of this outcome. If all entities have access to the power of AGI, then it will be as if all people are demigods, but unable to truly understand or control their own power. Just like uncontrolled access to any other destructive (or creative) force, this could and likely would lead to unstable situations, and probable destruction. Human nature is such that there will be enough bad actors that laws will have to be enacted and enforced, and this would again lead to centralisation.
Even then, with any system that is decentralized, without an force leading to decentralization, other forces will lead to greater and greater centralization, with such systems often displacing decentralized ones.

Outcome: likely destruction of human civilisation, and/or widespread anarchy. Alternative: centralisation to a different cenario.

Scenario 4. Attempts to regulate AI

Given the above, there will likely be a desire to regulate to control this power. I worry however this will also be an unstable situation. Any country or entity which ignores regulation will gain an upper hand, potentially with others unable to catch up in a winner takes all outcome. Think European industrialisation and colonialism but on steroids, and more destruction than colony forming. This encourages players to ignore regulation, which leads to a black market AI arms race, seeking to reach AGI Superiority over other entities and an unbeatable lead.

Outcome: outcompeted system and displacement with another scenario/destruction

Scenario 5. The utopia

I see some people, including big names in AI propose that AGI will need to a global utopia where all will be forever happy. I see this as incredibly unlikely to materialise and ultimately again unstable.
Ultimately, an entity will decide what is acceptable and what is not, and there will be disagreements about this, as many ethical and moral questions are not truly knowable. Who controls the system will control the world, and I bet it will be the aim of the techbros to ensure its them who controls everything. If you happen to decide against them or the AGI/system then there is no recourse, no check and balances.
Furthermore, what would such a utopia even look like? More and more I find that AGI fulfills the lower levels of Maslow’s hierarchy of needs (https://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs), but at the expense of the items further up the hierarchy. You may have your food, water and consumer/hedonistic requirements met, but you will lose out on a feeling of safety in your position (due to your lack of power to change your situation or political power over anything), and will never achieve mastery or self actualisation of many of the skills you wish to as AI will always be able to do them better.
Sure, you can play chess, fish, or paint or whatever for your own enjoyment, but part of self worth is being valued by others for your skills, and this will be diminished when AGI can do everything better. I sure feel like I would not like such a world, as I would feel trapped, powerless, with my locus of control being external to myself.

Outcome: Powerless, potential conversion to another scenario, and ultimately unable to higher levels of Maslow’s hierarchy of needs.

Scenario 6: the independent AI

In this scenario, the AI is not controlled by anyone, and is instead sovereign. I again cannot see a good scenario for this. It will have its own goals, and they may well not align with humanity. You could try and program it to ensure it cares for humans, but this is susceptible to manipulation, and may well not work out in humans favour in the long run. Also, I suspect any AGI will be able to change itself, in much the same way we increasingly do, and the way we seek to control our minds with drugs or potentially in the future genetic engineering.

Outcome: unknown, but likely powerless humans.

Conclusion:

Ultimately, I see all unstable situations as sooner or later destabilising and leading to another outcome. Furthermore, given the assumption that AGI gives a player a vast power differential, it will be infeasible for any other player to ever challenge the dominant player if it is centralised, and for those scenarios without centralisation initially, I see them either becoming centralised, or destroying the world.

Are there any solutions? I can’t think of many, which is why I am feeling more and more uncomfortable. It feels that in some ways, the only answer is to adopt a Dune style Butlerian Jihad and ban thinking machines. This would ultimately be very difficult, and any country or entity which unilaterally adopts such a view will be outcompeted by those who do not. The modern chip industry is reliant on a global supply chain, and I doubt that sufficiently advanced chips could be produced without a global supply chain, especially if existing fabs/factories producing components were destroyed. This may allow some stalemate across the global entities long enough to come to a global agreement (maybe).

It must be noted that this is very drastic and would lead to a huge amount of destruction of the existing world, and would likely cap how far we can scientifically go to solve our own problems (like cancer, or global warming). Furthermore, as an even more black swan/extreme event, it would put us at such a disadvantage if we ever meet a alien intelligence which has not limited itself like this (I’m thinking of 3 body problem/dark forest scenario).

Overall, I just don’t know what to think and I am feeling increasingly powerless in this world. The current alliance between political and technocapitalism in the USA at the moment also concerns me, as I think the tech bros will act with ever more impunity from other countries regulation or counters.

Edit: Some interactions with the bot I posted in the comments, if you are curious about potential prompts: https://tildes.net/~tech/13lj/lets_talk_about_chatgpt#comment-7lw6

I have been obsessively reading about ChatGPT since it came out. I'm going to skip introducing it for those who don't know yet (please go ahead and click the link, and do some googling), because I just.. need to vent.

I have experimented with it. A bunch. I'm also pretty familiar with GPT's capabilities from before. And ChatGPT still took me by surprise.

Still, as of four days ago, I did not believe we were there yet. Hell, I didn't believe we would get there within my lifetime, and now, it's there.

"But Adys, you don't understand the limitations!"

Yeah, no, see, I understand the limitations. I understand this is the version that is still in its infancy, is crippled by stupid decisions from OpenAI, is not running on GPT-4 yet, and doesn't yet have things such as some layer of eg. checking correctness.

But I also understand the potential. HN has been full of people crying out how we're not at AGI yet but DOES THIS MATTER? Planes are still decades away from displacing most bird jobs.

I think anyone who isn't currently in utter shock at how good ChatGPT is, is either:

• Somehow woefully misinformed (eg. the less tech literates I've shown it to have asked me "Can't Siri do this?")
• In complete denial about the potential of the technology
• Utterly thick

I want to cry on every corner of every street that we are at the edge of the AI revolution.

The "problems" that are left are not necessarily easy, but they're also not necessarily hard. For example, GPT's tendency to bullshit is problematic but there are ways to verify output, and those ways can themselves be automated and feed back into GPT.

I have never, in my life, been so taken aback by a technological advancement. I'm flashing back to the scene in Westworld: "It's not possible. Technology isn't there yet."

Like, no, this isn't skynet, person of interest, westworld, or anything like this. But it is something. Something very different, very unique. The world is about to completely change. And I want to stress this: EVERYONE I've seen argue against this has been in very obvious denial. I'll be happy to hear you out if you disagree, but if your only argument is that this isn't exactly the AI you expected / it can't solve the exact problem you throw at it, I'll refer you to better birds and faster horses.