In this short piece, Chollet analogizes autoregressive, transformer-based, generative LLMs to program databases, and prompt-engineering to query optimization. I think this is a good analogy and it...
In this short piece, Chollet analogizes autoregressive, transformer-based, generative LLMs to program databases, and prompt-engineering to query optimization. I think this is a good analogy and it helps frame the finicky nature of fiddling with LLM prompts.
Prompt engineering is a short term problem. OpenAI is trained on data through September 2021. Eventually, the OpenAI data will presumably be trained on data that includes a metric fuck ton of...
Prompt engineering is a short term problem.
OpenAI is trained on data through September 2021.
Eventually, the OpenAI data will presumably be trained on data that includes a metric fuck ton of information on how to prompt engineer all different sorts of use cases.
In fact, I am kind of surprised no one has fine tuned an LLM to solve this problem.
OpenAI can only do so much. Prompt engineering at its core is about providing information to the LLM that it needs to do its job. Some prompt engineering is about doing that in a specific way that...
OpenAI can only do so much. Prompt engineering at its core is about providing information to the LLM that it needs to do its job. Some prompt engineering is about doing that in a specific way that the LLM wants it - and we've seen massive ergonomic improvements here. Back in the day, you could only get GPT-3 to do what you wanted with few-shot example based prompts, ex:
Please write a brand mission statement for each company:
```
Company: Ford
Motto: Built Ford Tough
Mission Statement: To put an F150 in every driveway and make children crossing the road invisible to drivers.
```
Company: Google
Motto: Don't be evil
Mission Statement: Organize the world's information.
```
Company: $company
Motto: $motto
Mission Statement:
And then GPT-3 starts writing the mission statement at the end of the prompt.
That's how we used to do this a couple of years ago. Then OpenAI released davinci-instruct-00n and prompt engineers were blown away. What used to take few-shot now was zero-shot. You just needed to describe what the AI needed to do and give no examples for good results (and I've seen the numbers, it really was as good or better as far as users were concerned). My speculation is that OpenAI used API request information to build a corpus of instruction-to-output mappings. The example above, if properly integrated into the model, would let you input "Please write a brand mission statement for this company: $company. It's motto is $motto." And they had hundreds of millions of requests (at least) in this format to train from. The structure in the prompt above is straight from an example OpenAI provided in the GPT-3 playground, so many startups used the structure exactly, making it easier to train on.
But today we still have prompt engineering, it's just the quality bar has gone up. There are still many decisions to be made. Do you want to include the motto? Just the company name isn't enough, but of the billions of pieces of information you could include what will work best and why? Include everything and you kill the signal-to-noise ratio (and run out of context window) - that kind of tradeoff calculation is a classic engineering problem.
Edit: And this doesn't even get into multi-stage prompting. That's straight-up natural language programming.
I sort of see it the other way around—fine-tuning is a short-term solution, and actually understanding the "program space" of a model and selecting an optimal program to solve your problem is much...
Prompt engineering is a short term problem.
In fact, I am kind of surprised no one has fine tuned an LLM to solve this problem.
I sort of see it the other way around—fine-tuning is a short-term solution, and actually understanding the "program space" of a model and selecting an optimal program to solve your problem is much more valuable than spending the resources to tune a foundation model on a narrower problem space. I would think the more generalized a model is, the less fine-tuning should be necessary.
From a professional perspective, I'm interested to see what can be done with Prompt Engineering and Kimball data models. If an NLP layer can take inputs, figure out the most optimal data model to...
From a professional perspective, I'm interested to see what can be done with Prompt Engineering and Kimball data models.
If an NLP layer can take inputs, figure out the most optimal data model to construct based on time frequency of usage.... you've got some very quick and tasty Models getting thrown out that help many.
The way I've been thinking about it, at least in relation to image generators, is that the set of all possible images already exists in some multidimensional space. The process of turning a text...
The way I've been thinking about it, at least in relation to image generators, is that the set of all possible images already exists in some multidimensional space. The process of turning a text caption into vectors that correspond to a given image is like plotting a set of coordinates to a particular area in this space, with images that share a certain style, composition, or subject matter all grouped close together. You can sort of see this in videos that "explore the latent space" by making random-walk adjustments to parameters for a given generation, causing it to slowly morph between countless similar variations, every one of which could be a discrete image.
Incidentally, this helps shed some light on what's really going on with the training process. The popular conception is that these models simply average the results of the training data, or paste together parts of it wholesale, Frankenstein-style. The reality is more like the training process plots real-world image-caption pairs onto the higher-dimensional "map" of the latent space, so that when given a new caption, the model can easily jump to that conceptual area and "discover" a novel image that fits the description. Like a scatterplot that uses thousands of datapoints to create a line of best fit.
We flatter ourselves to think that we create art and music from nothing, but technically speaking there are a very large but finite number of possible melodies, paragraphs, and arrangements of pixels in a 1024x1024 grid. It's just that most of these possible combinations are garbage, and up until now it's taken the creative equivalent of hacking through a tangled jungle to find the treasure, a process that takes years of learning and hard work and luck. Now we've got a technology that can spot the gold from space and fly you there in a matter of seconds, a magic shortcut unlocked with the right incantation. This is a boon for creative brainstorming and image fashioning, but not so great for folks that make a living as rainforest guides. (My solution? Require any treasures found with this shortcut to be displayed in the public museum instead of sold on the open market.)
In this short piece, Chollet analogizes autoregressive, transformer-based, generative LLMs to program databases, and prompt-engineering to query optimization. I think this is a good analogy and it helps frame the finicky nature of fiddling with LLM prompts.
Prompt engineering is a short term problem.
OpenAI is trained on data through September 2021.
Eventually, the OpenAI data will presumably be trained on data that includes a metric fuck ton of information on how to prompt engineer all different sorts of use cases.
In fact, I am kind of surprised no one has fine tuned an LLM to solve this problem.
OpenAI can only do so much. Prompt engineering at its core is about providing information to the LLM that it needs to do its job. Some prompt engineering is about doing that in a specific way that the LLM wants it - and we've seen massive ergonomic improvements here. Back in the day, you could only get GPT-3 to do what you wanted with few-shot example based prompts, ex:
And then GPT-3 starts writing the mission statement at the end of the prompt.
That's how we used to do this a couple of years ago. Then OpenAI released
davinci-instruct-00nand prompt engineers were blown away. What used to take few-shot now was zero-shot. You just needed to describe what the AI needed to do and give no examples for good results (and I've seen the numbers, it really was as good or better as far as users were concerned). My speculation is that OpenAI used API request information to build a corpus of instruction-to-output mappings. The example above, if properly integrated into the model, would let you input "Please write a brand mission statement for this company: $company. It's motto is $motto." And they had hundreds of millions of requests (at least) in this format to train from. The structure in the prompt above is straight from an example OpenAI provided in the GPT-3 playground, so many startups used the structure exactly, making it easier to train on.But today we still have prompt engineering, it's just the quality bar has gone up. There are still many decisions to be made. Do you want to include the motto? Just the company name isn't enough, but of the billions of pieces of information you could include what will work best and why? Include everything and you kill the signal-to-noise ratio (and run out of context window) - that kind of tradeoff calculation is a classic engineering problem.
Edit: And this doesn't even get into multi-stage prompting. That's straight-up natural language programming.
I sort of see it the other way around—fine-tuning is a short-term solution, and actually understanding the "program space" of a model and selecting an optimal program to solve your problem is much more valuable than spending the resources to tune a foundation model on a narrower problem space. I would think the more generalized a model is, the less fine-tuning should be necessary.
From a professional perspective, I'm interested to see what can be done with Prompt Engineering and Kimball data models.
If an NLP layer can take inputs, figure out the most optimal data model to construct based on time frequency of usage.... you've got some very quick and tasty Models getting thrown out that help many.
The way I've been thinking about it, at least in relation to image generators, is that the set of all possible images already exists in some multidimensional space. The process of turning a text caption into vectors that correspond to a given image is like plotting a set of coordinates to a particular area in this space, with images that share a certain style, composition, or subject matter all grouped close together. You can sort of see this in videos that "explore the latent space" by making random-walk adjustments to parameters for a given generation, causing it to slowly morph between countless similar variations, every one of which could be a discrete image.
Incidentally, this helps shed some light on what's really going on with the training process. The popular conception is that these models simply average the results of the training data, or paste together parts of it wholesale, Frankenstein-style. The reality is more like the training process plots real-world image-caption pairs onto the higher-dimensional "map" of the latent space, so that when given a new caption, the model can easily jump to that conceptual area and "discover" a novel image that fits the description. Like a scatterplot that uses thousands of datapoints to create a line of best fit.
We flatter ourselves to think that we create art and music from nothing, but technically speaking there are a very large but finite number of possible melodies, paragraphs, and arrangements of pixels in a 1024x1024 grid. It's just that most of these possible combinations are garbage, and up until now it's taken the creative equivalent of hacking through a tangled jungle to find the treasure, a process that takes years of learning and hard work and luck. Now we've got a technology that can spot the gold from space and fly you there in a matter of seconds, a magic shortcut unlocked with the right incantation. This is a boon for creative brainstorming and image fashioning, but not so great for folks that make a living as rainforest guides. (My solution? Require any treasures found with this shortcut to be displayed in the public museum instead of sold on the open market.)