15
votes
Funny output from Claude.ai: a codeblock in a question about an early rennaissance theological work
I asked Claude for references to some ideas the ai had output earlier about a 16th century cardinal (Sadoleto). It's response was, Let me analyze the letter, and then this:
async function searchForSadoletoLetter() {
try {
const files = ['sadoleto_letter.txt', 'sadoleto.txt', 'sadoleto_to_geneva.txt', 'letter_to_geneva.txt'];
for (const filename of files) {
try {
const content = await window.fs.readFile(filename, { encoding: 'utf8' });
console.log(`Found file: ${filename}`);
console.log("First 500 characters:");
console.log(content.substring(0, 500));
return content;
} catch (err) {
// File not found, try the next one
}
}
console.log("Could not find any files containing Sadoleto's letter");
return null;
} catch (err) {
console.error("Error searching for files:", err);
return null;
}
}
const content = await searchForSadoletoLetter();
// Since we don't have the letter available as a file, let's search for published versions online
console.log("The letter isn't available as a file. We would need to rely on scholarly knowledge about Sadoleto's letter to Geneva (1539).");
console.log("Sadoleto's letter was published in 'A Reformation Debate: Sadoleto's Letter to the Genevans and Calvin's Reply' (ed. John C. Olin).");
This strikes me as odd? Did Claude write a script to generate its own response? Have coders put something like this in as a gaurdrail?
edit: details about earlier convo
It's important to remember what LLMs are: incredibly good autocomplete. While they are fundamentally different to your phone's autocomplete, they still do the same job: guess the next word (token, but whatever) over and over to make a response.
Like with your phone, tweaks can be made to what next word predictions might be. Claude is often used for generating code, so during training answers with code may get rated higher in more situations, leading to more code appearing in its outputs.
The most obvious thing to me here, as a developer, is in the code in the response itself.
What I'm guessing happened was that the marker for starting a code block got high enough in the probability ranking that it got chosen as the next "word", and since the previous part of the response is part of the input to getting the next "word" it started writing code because that's what goes in a code block. Since it was stuck in code mode, it eventually output a hard-coded reference as a way of fulfilling the prompt, albeit in an unusual way due to what came before in the response.
Hah this exact thing happened when I asked LeChat:
And it started calculating a weird bastardisation of an odds-ratio analysis with randomly generated percentile chances??? 😂
Like alright fair if the LLM was gonna poop out some sort of statistical reasoning considering I said the word "odds", but the context here was me wanting to get some quick info about some internet drama LOL.
That's hilarious lol it gave you just random odds using the answer to life, the Universe, and everything as the seed. Looks like you've got a shitposter on your hands haha
People have been training models for contexts where they do have access to code interpreters to do calculations for them. And they have also seen a lot of blog posts and things that say "Now we need to fetch the doobledo" and then break into code to do it.
They also don't have great mechanisms for retrieving things from memory in the way a human would. Was the stuff you were asking about that had been output earlier actually in the same chat? Had it been pushed out of the context? The model might not actually have a way to get that bird to come when called, so to speak, and is (luckily?) imagining that it has a way to recall that information instead of imagining the information from whole cloth.