From Karpathy's blog post: Also see: The space of minds Recently there have been news stories about how some people can become possessed by ghosts!
From Karpathy's blog post:
Stated plainly, today's frontier LLM research is not about building animals. It is about summoning ghosts. You can think of ghosts as a fundamentally different kind of point in the space of possible intelligences. They are muddled by humanity. Thoroughly engineered by it. They are these imperfect replicas, a kind of statistical distillation of humanity's documents with some sprinkle on top. They are not platonically bitter lesson pilled, but they are perhaps "practically" bitter lesson pilled, at least compared to a lot of what came before. It seems possibly to me that over time, we can further finetune our ghosts more and more in the direction of animals; That it's not so much a fundamental incompatibility but a matter of initialization in the intelligence space. But it's also quite possible that they diverge even further and end up permanently different, un-animal-like, but still incredibly helpful and properly world-altering. It's possible that ghosts:animals :: planes:birds.
LLMs are humanity's "first contact" with non-animal intelligence. Except it's muddled and confusing because they are still rooted within it by reflexively digesting human artifacts, which is why I attempted to give it a different name earlier (ghosts/spirits or whatever). People who build good internal models of this new intelligent entity will be better equipped to reason about it today and predict features of it in the future. People who don't will be stuck thinking about it incorrectly like an animal.
Recently there have been news stories about how some people can become possessed by ghosts!
Could someone explain what “bitter lesson pilled” means. The explanation of something arranged so that it benefits from additional computations for free is unclear to mean when the writer uses the...
Could someone explain what “bitter lesson pilled” means. The explanation of something arranged so that it benefits from additional computations for free is unclear to mean when the writer uses the term later. I also have no understanding of how the words bitter lesson pilled conceptually would mean that.
The "bitter lesson" for AI researchers is that after LLM's were invented, a lot of their fancy, special-purpose algorithms became obsolete because it turned out that using more compute with a...
The "bitter lesson" for AI researchers is that after LLM's were invented, a lot of their fancy, special-purpose algorithms became obsolete because it turned out that using more compute with a general-purpose algorithm could do a better job of solving the problems they were working on. So they had to switch to researching something else.
So, "bitter lesson pilled" describes the mindset of an AI researcher who keeps that in mind and tries to do research that will hopefully be useful now that there are LLM's to compete with. It might be an alternative approach to LLM's, but it needs to be both fairly general-purpose and scale up well.
(It seems like pretty awkward Internet jargon for doing things that scale well.)
Computing is of course not free at the scale that the AI labs are using it, but it's more abundant than it used to be. They have much more powerful hardware.
From Karpathy's blog post:
Also see: The space of minds
Recently there have been news stories about how some people can become possessed by ghosts!
Could someone explain what “bitter lesson pilled” means. The explanation of something arranged so that it benefits from additional computations for free is unclear to mean when the writer uses the term later. I also have no understanding of how the words bitter lesson pilled conceptually would mean that.
The "bitter lesson" for AI researchers is that after LLM's were invented, a lot of their fancy, special-purpose algorithms became obsolete because it turned out that using more compute with a general-purpose algorithm could do a better job of solving the problems they were working on. So they had to switch to researching something else.
So, "bitter lesson pilled" describes the mindset of an AI researcher who keeps that in mind and tries to do research that will hopefully be useful now that there are LLM's to compete with. It might be an alternative approach to LLM's, but it needs to be both fairly general-purpose and scale up well.
(It seems like pretty awkward Internet jargon for doing things that scale well.)
Computing is of course not free at the scale that the AI labs are using it, but it's more abundant than it used to be. They have much more powerful hardware.