An AI researcher writes about his crisis of faith

From the blog post:

Even if the nightmare scenario of full human redundance is not imminent, the fact that a considerable number of people claim that this is their goal is unsettling. This view is perhaps most succinctly stated by the startup Mechanize, which aims to “fully automate the economy”. As a participant in the economy, I find this goal deeply objectionable. I think that trying to fully replace all human labor is not a morally acceptable goal to have. It’s frankly horrifying. To be clear, “fully automate the economy” means “destroy society as we know it”. Unfortunately, similar goals are articulated by quite a few inside and outside Silicon Valley. I think the callousness of such objectives should be called out whenever they are encountered.

The great irony of this is that those who claim to want to automate all human labor are typically the kind of competitive, smart, high-agency persons who absolutely need to have a purpose and something to build. They would hate to be redundant. Yet, here they fly, like so many moths to a flame.

[...]

Why didn't I just quit AI research? My predicament would seem like that of a vegan butcher, or a monk who makes money on Onlyfans. But me quitting and becoming an Uber driver would not make the world better. The pace of AI progress would clearly not slow down noticeably. And I assure you, I still love AI research, even if I sometimes hate what AI does to the world. I'm not even very good at anything else. AI is what I do. And I think that I can do more good by trying to steer my field in a good direction than if I became an Uber driver. So I’d rather think of myself as akin to a hypochondriac doctor, or a pilot with a fear of heights.

[...]

From this perspective, the history of AI is a history of attempts to mimic the specific combination of behaviors and capabilities that humans have; most of them successful in some way, but all of them quite different to humans. The onslaught of LLMs becomes a push in a particular capability direction. Understanding that direction becomes crucial to figuring out which types of human intellectual patterns and capabilities will become more important in the future. Where the new domains of human excellence will appear. But this understanding can also help us develop different types of AI that are more complementary to what humans can do and like to do. Seeing intelligence as a scalar, where machines can overtake humans, is a recipe for paralysis; dissolving this faulty notion gives us the freedom to act. More of this argument in the article I linked above; what's important here is that there are things that can be done. Indeed, there's a lot to do. Such as building mechanisms for meaningfully incorporating humans in creative search processes, and open-ended learning and discovery processes that are not based on imitating what humans do.

But not everything has a technical fix. Norms, structures, and laws are probably more important. I still think that trying to automate humans out of the processes that give them, and our civilization, meaning is immoral. We need to build counter-narratives, and be vocal that "fully automating the economy" is not an acceptable goal to work towards. We also need to push hard to counter the centralization of power that so easily comes with lavishly funded tech companies trying to achieve monopolies on some layers of the AI stack. Our best bet for a future where we all matter is one with a myriad different forms of intelligence, open and accessible for all to use as tools for our natural intelligences. Let's get to work.