The magic is in the language
In all the hype and debate and excitement and frustration around AI, I think sometimes the fact that we've achieved something really cool gets lost. I want to talk about that a bit, which I'll try to do in a simple way, without too much math or computer science.
There's a surprising amount of intelligence encoded in applied language. To me that's the most remarkable insight in LLM technology. Except it's not really surprising at all. The whole point of language is to encode intelligence. Communication, at its core, is about sharing intelligence. LLMs agents are a way of turning the intelligence encoded in language, particularly written language, into useful work.
You might imagine that language is just the starting point when building LLMs. That it gets converted into computer stuff and then it's not language anymore. But in a way it's language all the way down. Distill language into math and then run gradient descent on it and the intelligence (for lack of a better term) bubbles out.
Forgive that hyperbole, but there is something a little bit magical about it, and the technology is entirely unmagical. The magic is in the language.
At the core the tech is pretty simple. First you go and get all of the language. Just completely idiotic amounts of applied language. Because there's a threshold: if you don't use enough language you don't get a useful result. The more language you use, assuming it's good quality, the more useful the result.
After you have all of the language, you convert it into tokens, which are essentially pieces of words represented as numbers. You also compute a compressed version of the relationships between the tokens. The frontier labs don't say how much this costs publicly anymore but it's safe to say hundreds of millions in compute are required for this process. Each time you do it.
Once you've determined the token relationships, you've pre trained an LLM. You have the initial weights. At this point your LLM can output credible prose pretty reliably when you run inference on the weights. Meaning that you can take a given set of language tokens (a prompt) and determine which token should come next, over and over again, until you have a response. The fluency of the prose can be uncanny, and there are latent tendencies towards useful output, but you can't really do much with it yet.
The next step, fine tuning, modifies the weights in order to nudge the inference towards useful things like instruction following, tool use and reasoning. Or a simulacrum of reasoning. Fine tuning is a massive, multi step, iterative process that turns your LLM into a useful tool that can do more than output believable prose.
Something that's really fascinating to me about fine tuning is that what you're training is still the language relationship weights, and what comes out during inference is still language. But now it's language that is more likely to result in useful output and behaviors. Language is the substrate that LLMs reason through. A model that's more likely to infer the words "but what if I look at it another way" is more likely to consider multiple possibilities. A model that reliably infers language around tool use in appropriate situations is an agent. The behavior is encoded in the language.
After tens of millions in fine tuning costs, assuming you got it right, you have the core of a LLM that can function as an AI agent. You can put it in a harness and ask it to do things in the language of your choice and it will actually be able to do them a shockingly high percentage of the time. It can even do things that weren't explicitly part of the training. That's something which has never happened before outside of science fiction. Until recently it was one of those computer sciencey things that was probably going to happen someday, after we were all dead.
It sucks that this tech revolution is being driven by the capital class, and that it's happening as a mad dash for market domination and golden IPOs. In a more perfect world technology derived from collective human intelligence would be a public good rather than a profit driver.
But nevertheless, the ability to imprint some part of human intelligence into an autonomous agent is a remarkable achievement. One that we can't yet see the full size of. And it's possible not because we typed instructions into an interface, but because we've spent centuries encoding our intelligence into language.
Its amazing how quickly we've gotten used to technology that would have been considered pretty far future science fiction in the early 2000s, and literal magic any time before the first half of the 20th century. Like if you showed a random person from any time before around 1950 a cell phone running ChatGPT in interactive mode, the only possible explanation virtually anyone would be able to come up with is that you captured a demon inside a smooth rock.
There wouldnt even be a possible conception of an explanation outside of a handful of extremely advanced cutting edge computer scientists and researchers. If you went back a few hundred years further, you'd be prosecuted for witchcraft almost immediately.
Its not like something like a car or engine or even an airplane where they might not fully understand the mechanisms at work, but they'd understand that it was something that technology could eventually achieve, it's something that intuitively just feels like the dark arts to the layperson, probably more than just about any technology we've developed that I can think of.
I feel like this underestimates folks from past eras, especially as late as 1950. You don't need to understand how something works to accept it.
The Mechanical Turk actually was a demon trapped in a smooth rock—well, technically a human trapped in a wooden cabinet. Close enough. But it was presented as technology: nothing but gears inside, no strings leading elsewhere, use this lodestone to see that it's not being controlled magnetically, etc. And it sounds like audiences in the 1700s and 1800s accepted that the intelligence was mechanical? There were definitely skeptics, but I get the sense that it was not obviously a hoax. Technology was a plausible explanation for the wondrous even back then.
Then again, I do see this quote on Wikipedia:
So maybe out of an abundance of caution, don't wave around your smartphone in front of just anyone the next time you visit the 1780s—get to know them first.
I was thinking recently that we’ve already caught up with the ship’s computer from Star Trek, but honestly we’ve probably surpassed it. Doing exactly what we often see in the show is almost mundane already: asking freeform questions about some topic that involves summarising a huge chunk of data that just came in and then cross referencing that with all the existing data across whatever systems the computer’s plugged into, regardless of format or consistency. Fucking Microsoft even does a respectable job of it as part of O365, the least exciting software suite in history!
But in reality I don’t even need to make the requests… computery sounding like they do in the show. No “search all databases for…” because that’s usually implicit; no slightly altering requests that sound suspiciously like boolean searches until the computer stops saying “no results found”; not even that many drill down questions or explicit calls for clarification. Often enough I’ll get a proactive “oh by the way, you might want to look out for X potential problem” on the end of a response that I genuinely hadn’t previously considered and may well have missed.
Ethics and overuse and hype and marketing bullshit and environmental impact and the rest are all real, serious problems. But for a baseline definition of “helps out at a level similar to a decent coworker, not a traditional computer program”, this tech works in a way that I would’ve called laughably implausible a decade ago. And I was already a decently experienced programmer with a background in machine learning a decade ago!
There is one big feature depicted in Star Trek that modern LLMs don't have but I want. The Star Trek computer has (and uses) a “database” of facts and knowledge. LLMs have everything baked into the weights. I want to be able to add to and correct information in my computer’s database without having to retrain a new model.
Not exactly true, the early ones worked like this and had huge hallucination problems. Nowadays they’re trained to search the web for facts instead of trusting their gut.
But I agree with your main point, having an actual auditable knowledge graph that’s integrated with the LLM but separate from it is going to be more and more important in the near future. Because we’ve been pretending that a web search is a way to find truth, but we all know the internet’s absolutely flooded with misinformation. Now bad actors are actively trying to exploit the gullibility of LLMs to mislead them in various ways. And they’re doing it at AI slop scale… you can’t trust everything you read online, but LLMs often do, so they’re right in the crosshairs of a bunch of nasty AIEO schemes.
I think that problem’s only going to get worse. Seems like the big vendors like Anthropic and OpenAI will soon need to maintain their own internal, curated Wikipedias and set up their agents as frontends to them. Nobody wants to be the world’s arbiters of truth though, it’s a thorny problem.
MCP servers are pretty much this, just a REST API and a standardised JSON file that tells the model how to traverse it. Assuming you hook it up to a reliable data source you get more or less that exact behaviour (not 100% perfect because there’s still that bit of nondeterminism in how it presents the data it parses from the API, but pretty damn close on anything that’s available in a clear way via whatever data sources you’re using).
I was actually thinking of exactly that when I said even Microsoft does a respectable job! If it’s properly set up on a corporate plan it’ll pull in data from Outlook, Teams, OneDrive, etc by default and has one click plugins to do the same with JIRA, Confluence, GitHub, and a bunch of others. It’s feels less like Star Trek when you’re asking it to find mundane procedural info that’s scattered across 15 different corporate docs within a sea of 10,000 others, but it is extremely useful, and we can comfort ourselves by thinking the Enterprise crew had to spend time looking up Federation HR policies and whatever else offscreen too! It very much wouldn’t be able to do that on data encoded into the weights alone.
If you want the “personal database” kind of behaviour, I’d prime it with the ability to query Wikipedia (bonus points for using an offline dump and a local model), and then tie it into your note taking app of choice as well so you have a channel to add personal data too. You can even just give it access to a Google Drive directory and put whatever you like in there.
Indeed. And it really is wild how quickly it starts to become normal. I'd have to stop and think about it for a while to count all of the things that happen on a given workday that would have been laughably implausible not very long ago.
I have a bone to pick with people that claim these programs don't "understand anything". What is understanding if not reverse-engineering a system through its inputs and outputs? The problem is these models present the same level of confidence regardless of the quantity of training data on a given subject. People that want to believe they're stupid and useless can focus on how the model made up a fact about their life (when they aren't enough of a celebrity for it to know anything about them) or their home town, or school etc. People that want to believe they are a cognition replacement extrapolate from the model's ability to write code 1000x better than they can (for those that are not professional programmers).
Yeah extreme interpretations both ways. It must be something we don't understand!
Just a couple of days ago I used to think that the models are basically "just" predicting the next token. I.e. if I feed it
"Happy birthday to" it is highly likely to output "you" as the next token. However I saw a short YouTube clip that made a persuasive argument that models are infact doing at least something more advanced. If I feed an LLM:
It will likely output:
3 001 138. The argument is that the LLM hasn't learned this is the likely answer to the sequence of input tokens. If that was the case they'd often produce garbage answers for this type of input (older and/or smaller models do struggle with this). But, since newer models often seem to get it right, it is in my view now more likely that they have some form of concept of addition embedded in their weights. So they don't "just" predict the next token, they do seem capable of mental arithmetic.Perhaps there's some other explanation. Perhaps this breaks down on larger input sequences. I do however find it intriguing.
Before fine tuning (which includes RL), next token prediction is a reasonable mental model, but fine tuning changes things quite a lot. You're right that there's a lot of capability embedded in the weights.
Even with just pre-training, you might be surprised by the response to your example from a large model because the training creates complex and layered associations that already look like something more than next token prediction. A large model with no RL or other post training steps might not get the answer right, but it would "know" that it was a math question and its "guess" probably wouldn't be terrible because even if it had never seen those particular numbers in a sequence, it has relationships between a lot of numbers, and the shape of a lot of math problems, in training.
Thank you for stating a position that's somewhere in the middle in terms of optimism. Something I wonder then is what new aspect of intelligence has LLM captured, compared to say old-fashioned coding. Maybe we could say a typical computer program is an expression of the strict logical relation of information, what sort of informational relationship does an LLM manifest? What does the training process distill from the data? Saying it's just detecting pattern is a bit too broad, since there's a sense that the pattern the LLM picked up is of some particular variety.
Language! Or more specifically the signal encoded in the relationship between bits of language after it's been used by humans to convey meaning. It's extracting traces of intelligence from our (mostly written) use of language.
I’m reminded of the quote where Feynman was talking about his work with a historian:
From this page. It also has a quote from Grothendieck that’s pretty good, too.
That’s not to say that all thought works that way. Chess players and musicians aren’t writing things down. But thinking by writing things down, writing and revising, is real thinking.
This reminds me of “A Man Without Words”, (see this absolutely fascinating Radio Lab episode about it https://radiolab.org/podcast/91725-words ).
The man in question, grew up without any language at all, and did not learn until he was well into adulthood. He recounts how learning language totally reshaped his thought processes, and brought to him an enhanced ability for planning and forethought that he never had prior.
It seems to suggest that language is not merely just a tool with which to communicate to people and describe the world, but actually a tool that enhances our ability to reason. So you may very well be correct that “the magic is in the language”.
The relationship between cognition and language deserves its own thread because there's a lot of debate. I think some version weak linguistic determinism is self evidently what's happening, but the relationship between language and thought is so complex, and so hard to study, that it will be an open question for a long time.
In general machine learning just learn the pattern in the data; a network can distinguish a cat from a dog because images of cats and dogs have distinct patterns. A LLM that is just trained on text can do stuff because there's patterns in the text.
That said AFAIK current LLMs also go through reinforcement learning, where the information doesn't necessarily comes from the data but is learned from "experience" in a simulated environment (e.g. AlphaGo), so they might be doing some "magic" that isn't in the training data.
That's true about RL, that's most of the fine tuning step. But mostly what's happening during that step is that the weights are being nudged to increase the strength of patterns that were already there.
And those patterns are intelligence encoded by humans in the course of communication.
There are some pretty good YouTube channels with good videos to help understand how LLMs work:
Recently I've been thinking about this quote from Wittgenstein, that understanding is essentially the correct dispatch of terms in context. Each context defines its own vocabulary, which means even the same words mean different things in different contexts, and differ from person to person. The way one learns to dispatch the terms in correct context is then, through practice and patterns. However this only leads to improvement only if there is a part of judgement, of what feedback deserves attention. This is where human bias, for all its strengths and weaknesses, show up: two people can intepret the same events differently (through their experiences and mindset) and end up understanding them in different ways.
LLMs try to proxy this "context" through the proximity and frequency of use with other words. In short, what meaning words have are not done through deliberation (like a dictionary/glossary) but rather an indexing of "this word is commonly used with these other words".
LLMs would be useful, if they have been trained on the context they would have been used in. That means it works on the generic queries, but on the domain specifics LLMs either will not have enough training data, or will contort the "same words" from its training corpus, forcing a language from an aggregate of contexts. This in my opinion, makes the communication aspect of language ever more unstable - it's some kind of subtle semantic satiation.
All in all, it's just my opinion, because my mental disposition makes me really want things clear cut whenever I embark on learning something. But there are people out there who thrive in jargon, buzzwords and ambiguity, and they make a lot of money from that too.
It's an interesting topic.
Assuming I'm reading this right, I want to clarify a couple points. One of the key features of LLM agents is that they can be useful even outside of the direct context they were trained in/on. They're more useful, for some things, when you're closer to the center of the training distribution but they can do something that looks like generalizing, some of the time, outside of the distribution
The reason for this appears to be that there's is more information encoded in the token relationships than you might imagine. One way to look at it is that words have both independent meaning and meaning that changes based on the context. Which means both that a word has self contained meaning that exists independently of any training data, and that dynamic meaning can be built on the fly out of word relationships as you write them. Because this is encoded in the weights, when you're running inference, meaning can survive into space that isn't specifically in the training. Even when there is no reference (so to speak) the model can blaze a path through the dark built out of token relationships and the meaning they encode. The words and the reasoning are the same thing, so they can go uncharted places.
It starts to fall apart at some point, but with a large enough model it can make it pretty far into the dark before that happens. That's especially true when it can generate feedback and incorporate the results into its inference, which is one of the defining characteristics of agents. The answer doesn't have to be in training for a model to find it as long as there's a way to test and iterate. In domains like code, math, biology and physics, deterministic testing is often easily accessible.