The thing I’m wondering is: how do you find the mistakes? I assume they did a few spot checks. Maybe look at every episode where it’s not the usual two speakers.
The thing I’m wondering is: how do you find the mistakes?
I assume they did a few spot checks. Maybe look at every episode where it’s not the usual two speakers.
I’m doing almost this exact project. Three things I do are: Manual spot checks based on heuristics, and also the interface I use for actually viewing the content has an integrated editor to...
I’m doing almost this exact project. Three things I do are:
Manual spot checks based on heuristics, and also the interface I use for actually viewing the content has an integrated editor to resolve issues on the spot. I and other people are active users of the data so feedback is plentiful.
Set models against each other to critique. I use Anthropic, GPT, and a fine tune of Llama. Disagreements get flagged for manual review.
Evolved prompts. When I have an error, I directly ask the model to consider the existing prompt, the erroneous text, the correct response, and give me a new prompt that will help detect and fix that kind of error in the future. I also rerun existing text against the new prompt.
Interesting! I am curious about the three adversarial models heuristic. It sounds similar to the Saturn 5 guidance computer. Do you still find situations where all 3 manifest the same error in the...
Interesting! I am curious about the three adversarial models heuristic. It sounds similar to the Saturn 5 guidance computer. Do you still find situations where all 3 manifest the same error in the same way? Essentially do you have any false negatives from this heuristic?
Definitely get false negatives, and more than a few false positives (especially on the Llama model). Sometimes there's just no way for a model to have sufficient context to understand what's going...
Definitely get false negatives, and more than a few false positives (especially on the Llama model). Sometimes there's just no way for a model to have sufficient context to understand what's going on -- for example, I have a fairly large set of text involving a Brazilian guy speaking English with a thick accent using sport-specific figures of speech that are being literally translated from Portuguese to English that have no natural analogues.
The thing I’m wondering is: how do you find the mistakes?
I assume they did a few spot checks. Maybe look at every episode where it’s not the usual two speakers.
I’m doing almost this exact project. Three things I do are:
Manual spot checks based on heuristics, and also the interface I use for actually viewing the content has an integrated editor to resolve issues on the spot. I and other people are active users of the data so feedback is plentiful.
Set models against each other to critique. I use Anthropic, GPT, and a fine tune of Llama. Disagreements get flagged for manual review.
Evolved prompts. When I have an error, I directly ask the model to consider the existing prompt, the erroneous text, the correct response, and give me a new prompt that will help detect and fix that kind of error in the future. I also rerun existing text against the new prompt.
Interesting! I am curious about the three adversarial models heuristic. It sounds similar to the Saturn 5 guidance computer. Do you still find situations where all 3 manifest the same error in the same way? Essentially do you have any false negatives from this heuristic?
Definitely get false negatives, and more than a few false positives (especially on the Llama model). Sometimes there's just no way for a model to have sufficient context to understand what's going on -- for example, I have a fairly large set of text involving a Brazilian guy speaking English with a thick accent using sport-specific figures of speech that are being literally translated from Portuguese to English that have no natural analogues.