This is counterintuitive because we know that word choices matter when we write, so how can choosing a different word not matter? Surely there are subtle grades of meaning to consider when picking...
This is counterintuitive because we know that word choices matter when we write, so how can choosing a different word not matter? Surely there are subtle grades of meaning to consider when picking one synonym over another?
That's going to depend on what words come before. In some positions, the next word will be highly constrained so it will always pick the next word the same way. There won't be any randomness at that position. At other places, the next word is a freer choice.
This won't prevent the AI labs from working on improving AI writing style. As training gets better, LLM's will become more sensitive about word choices and less likely to choose randomly. So, the amount of freedom delegated to the random number generator will drop. Probably not to zero, though!
Anthropic quietly announced a week ago they were rolling out watermarking to comply with the EU Code of Practice. Since they don’t want to have to differentiate traffic sources, the marginal cost is zero, and watermarking is pro-social, this will apply to everyone. They then offered an FAQ of how it works.
[...]
The entire practical effect is: There will be an API that will tell you if a given piece of writing comes from Claude. That’s it. And yet.
[...]
The rest of this post is about exploring why people are Big Mad about this, in large part as a worked example of how people get worked up over approximately nothing.
I wonder if one reason why people might seem angry with Anthropic's watermarking specifically is that Claude's specific writing style is so idiosyncratic and grating, to the point of being like a...
I wonder if one reason why people might seem angry with Anthropic's watermarking specifically is that Claude's specific writing style is so idiosyncratic and grating, to the point of being like a human-discernible watermark, and is so difficult to reliably change, that they assume, perhaps even unconsciously, the watermarking must have something to do with it.
What does make me angry is that, the last time I checked, Google's "public detector" for their watermarking is not a public detector, it's a private detector that their public LLM is allowed to use. You are forced to use their LLM chat, hope that their LLM decides to run the detector given your prompt, and then interpret its answer, along with whatever Google-perspective moralizing it added.
It's a real pattern, and you're right to point it out. The honest assessment is that code is genuinely load-bearing. Great! I have what I need. Here's the truth... and on and on and on I hate...
It's a real pattern, and you're right to point it out.
The honest assessment is that code is genuinely load-bearing.
Great! I have what I need. Here's the truth...
and on and on and on
I hate these machines and I hate that they wear the faces of humans and I hate that I am required to use them at work :(
I use blast radius frequently when discussing the, well, blast radius of a feature or fix. It's a good term and useful shorthand for the connectivity of a system. I'll stop using it when I'm dead...
I use blast radius frequently when discussing the, well, blast radius of a feature or fix. It's a good term and useful shorthand for the connectivity of a system. I'll stop using it when I'm dead or retired.
It is, but it is somehow the way that Claude weave it into its babble that makes it sound so much worse. Like Mmm Bop 29 years ago. Fine once in awhile. Less fine when it was every 3rd song on the...
It is, but it is somehow the way that Claude weave it into its babble that makes it sound so much worse. Like Mmm Bop 29 years ago. Fine once in awhile. Less fine when it was every 3rd song on the radio for a year.
I think you're right. The claudisms are really annoying and while this watermarking is something different, it's easy to confuse. I've been using this skill, it's pretty solid:...
I think you're right. The claudisms are really annoying and while this watermarking is something different, it's easy to confuse.
Here are Scott Aaronson's comments. So if the writing quality isn't that great to begin with, this doesn't make it worse. Any of the samples would use a similar number of AI catch-phrases. ...
[C]ontrary to many people’s intuitions, there’s no inherent tradeoff between watermarking and the quality of LLM output. Basically, nearly every LLM output was already a sample from a cloud of exponentially many possibilities, all of them about equally good, so there’s plenty of room to steer within that cloud without affecting anything that an ordinary user would notice.
So if the writing quality isn't that great to begin with, this doesn't make it worse. Any of the samples would use a similar number of AI catch-phrases.
I could add that, in addition, there’s recent progress (see here for example) on what I’ve called “semantic watermarking,” or watermarking at the level of the underlying concept vectors rather than the tokens themselves. This actually seems to work, albeit with no theoretical guarantees, and will hopefully make removing watermarks a lot harder—although the Barak et al. impossibility result suggests that under plausible assumptions, no LLM watermarking method will be completely foolproof.
...
Nevertheless, Google DeepMind implemented something very similar to my proposal in its SynthID, deployed in all its Gemini text models. But they heavily restricted who gets to detect the watermark, which made their admirable decision of limited use to my academic colleagues, who’ve been begging me for a way to detect whether their students are using AI to cheat. (For now, I mainly send them to Pangram, a leading AI detector not based on watermarking, as a first line of defense.)
And now, apparently to comply with EU regulations, Anthropic says they’ve deployed a watermarking scheme like mine where anyone will be able to do detection (though they also say in their FAQ that they’re still working on the detection API). Even OpenAI suggests that it plans to follow suit. So, four years after I seriously thought about this, it looks to my surprise like this is actually happening. Thanks, EU!
I think that might be read in a way that's uncharitable to LLMs and/or the method. Even if writing quality was good. There's still arguably a lot of variation in how any given human could write...
So if the writing quality isn't that great to begin with, this doesn't make it worse.
I think that might be read in a way that's uncharitable to LLMs and/or the method. Even if writing quality was good. There's still arguably a lot of variation in how any given human could write any given piece of text, depending on minute factors that basically boil down to pseudo randomness. Or in other words, even best-human level performance on the task of writing general text carries enough randomness that a perfect machine replica of it allows for this method at no loss of output quality. Wherever output quality of LLMs is currently lagging to the degree that humans use a lower-entropy next-token distribution (i.e. are more confident about the next word - let's just say in poetry. Also, this is a proper subset of "LLMs lag human performance"), there this watermarking mechanism will stop applying if/when models catch up to humans.
It's really quite ingenious. I lowkey fear what some people will do with the results of this watermark, mostly in over-interpreting that a text is AI to mean that the thought behind it is too. And/or applying the slop label by default, which is also common. But I think it's still a net-positive move.
I don't understand how anyone with a basic understanding of LLMs and/or randomness can believe this would affect output in any way. All they've changed is that the randomness is seeded and they...
I don't understand how anyone with a basic understanding of LLMs and/or randomness can believe this would affect output in any way. All they've changed is that the randomness is seeded and they keep the seed to reference later. This has to be a case of people talking about tech while lacking literacy in it, right?
Anyway, this sounds objectively good. I'd prefer if all AI companies spontaneously combusted, but I'm glad this is now a law. But I'm a bit dubitative about how this will actually work. How can it check text against the seed without having the prompt that generated it? If the chatbot's owners are the one providing the checking API, how do I trust that they're not just lying? How much does it actually take for 'human-edited' output to lose enough of its watermark to be indistinguishable from actual human written text? There's a lot of vagueness I'm worried about here.
Anthropic links to this paper. Apparently the tokens are chosen to score highly according to a random scoring function. This is not very clear to me, but here’s the key bit: Google’s AI summary...
Anthropic links to this paper. Apparently the tokens are chosen to score highly according to a random scoring function. This is not very clear to me, but here’s the key bit:
The key idea of Tournament sampling is to use a tournament-like process to choose an output token that scores highly with respect to some random watermarking functions. An illustration is given in Fig. 2 (top). First, we take the random seed rt provided by the random seed generator. This seed is passed to m (in this case, m = 3) watermarking functions g1, g2, g3, …, gm—these are independent pseudorandom number functions that assign a score gℓ(xt, rt) (in this case, a 0 or 1) to any candidate token xt ∈ V.
In the second stage (Fig. 2, bottom), we start by sampling M = 2m candidate tokens from the LLM distribution pLM(⋅∣x<t) (some tokens may appear multiple times): these are the initial participants of the m-layer tournament. We randomly divide these candidates into M/2 pairs, and, in the first tournament layer, in each pair the token with the higher score under g1(⋅, rt) is selected, and the other discarded (any ties are broken randomly). The remaining M/2 tokens are regrouped randomly into M/4 pairs, and the function g2(⋅, rt) determines the winners for this second tournament layer. This iterative process continues until one token emerges as the final winner, which becomes the output token xt. A formal description of Tournament sampling is given in Algorithm 2 in Methods.
By design, Tournament sampling selects a token from the LLM distribution that is likely to score higher under the random watermarking functions g1(⋅, rt), …, gm(⋅, rt). To detect whether a piece of text x = x1, …, xT is watermarked, we measure how highly x scores with respect to these functions.
Google’s AI summary seems a bit clearer, so maybe the thing to do is to ask your friendly AI what this all means?
It sounds like they start with the scoring functions and produce the text specifically so it scores high. So, they only need the scoring functions to verify the text. But they do need to keep them secret to prevent forgery, so we will have to rely on their API to tell us whether the text was from their LLM or not. Yeah, that’s not ideal.
Likewise, if I suspect a given piece of writing is AI generated, but the checking tools are closed source, does that now mean I need to check the writing against Claude and ChatGPT and Gemini and...
If the chatbot's owners are the one providing the checking API, how do I trust that they're not just lying?
Likewise, if I suspect a given piece of writing is AI generated, but the checking tools are closed source, does that now mean I need to check the writing against Claude and ChatGPT and Gemini and DeepSeek and etc etc? Each of them are competing, so I don’t see them working together to build a unified “check here” single combined platform unless they’re required by law.
This was the interesting quote to me, and I do think that makes it clearer what it is doing, rather than calling it a "watermark." I'm also fairly "AI-luddite" and don't use them hardly at all,...
Jai: The concept we want is "steganographic signature" but most people won't understand it.
This was the interesting quote to me, and I do think that makes it clearer what it is doing, rather than calling it a "watermark."
I'm also fairly "AI-luddite" and don't use them hardly at all, aside from boilerplate code (since my work is pushing us to use AI for ~something~), so this change likely wouldn't affect me at all.
Interestingly enough, I saw this after reading 404 Media's Anthropics text watermarking proves ai companies do not care at all about writing, which is a different take on the watermarking, from a...
But I think it's interesting because it seems like 404 media is generally against AI use for creative works, but then is also seemingly against this watermarking that would let it be more detectable?
It's a weird article, and it suggests to me that the author fundamentally misunderstood how LLM technology actually works up until the watermark thing forced him to look into it. Strangely, he...
It's a weird article, and it suggests to me that the author fundamentally misunderstood how LLM technology actually works up until the watermark thing forced him to look into it.
Strangely, he still seems to think that Claude and other LLMs use some completely different, non-LLM method to generate code. It's true that an LLM's code output is going to be more constrained than, say, its poetry output, but that's because LLMs are trained on code examples that are written in a highly constrained way and poetry examples that are written in a far less constrained way — because the level of constraint is a pretty inherent difference between code and poetry.
This means the watermark would be easier to detect for LLM-written poetry than for more highly regimented text, like a legal document where there just isn't much room for randomness. It has little to do with how much AI companies care about generating legalese versus poetry.
If you crank up the randomness on an LLM, you start getting some really weird gibberish that's a mixture of code, creative writing, technical documentation, slang, etc., etc. because the LLM doesn't actually "know" that these are all different kinds of text. They all go into the same training pot.
I did not read the 404 article so I may be misunderstanding you but there seems like there is still a great deal of decision space for an LLM to use to embed a signature while writing code when...
I did not read the 404 article so I may be misunderstanding you but there seems like there is still a great deal of decision space for an LLM to use to embed a signature while writing code when taking into consideration just things like white space, comments, bracketing styles etc. And then there are lots of different ways to write code that will compile down to the same or very similar machine/byte code. I do wonder how well it will survive e.g. linting but I imagine the intended purpose is mostly for written language rather than code.
According to Anthropic, there will be less watermarking in code, but you're right that it will show up more strongly in places where there's more leeway in which tokens the LLM can choose, like...
According to Anthropic, there will be less watermarking in code, but you're right that it will show up more strongly in places where there's more leeway in which tokens the LLM can choose, like comments:
As we noted above, AI watermarking takes advantage of decisions where either choice of a word would be equally good. Where an exact output is required—where there isn’t a choice, and something would be factually wrong or a piece of code would break if a different term was chosen—the watermark isn’t applied.
For example, once the model has written “2 + 2 =”, there is a very clear best choice for the next token (if the model is completing the sum, there isn’t an answer that’s equally as good as “4”; if it’s talking about George Orwell’s Nineteen Eighty-Four, there isn’t an answer that’s equally as good as “5”). The “nudge” of the watermark wouldn’t be applied here. For the same reason, code—which in very many cases has to be exact—has generally less watermarking than some other forms of text.
Having said that, in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code. But by definition, it will have a negligible effect on the actual code produced.
I imagine that the more strict the syntax has to be (e.g., if a company requires a very uniform code style), that harder it will be to differentiate Claude-generated code from human-written code. Which makes sense, because these constraints also make it harder to tell one human writer from another.
I'm curious how this will affect stuff like class and function names. A long name like AbstractWidgetFactory consists of multiple tokens, which might be long enough to start to see watermark...
I'm curious how this will affect stuff like class and function names.
A long name like AbstractWidgetFactory consists of multiple tokens, which might be long enough to start to see watermark effects. What do you call the class that coordinates them? Maybe the LLM says "Manager" and "Repository" are equally likely, but given the last few tokens, the watermark algorithm biases towards AbstractWidgetFactoryManager.
Now you've got a watermarked sequence of tokens that you have to repeat verbatim everywhere, regardless of whether you're AI or not:
AbstractWidgetFactoryManager awfm = new AbstractWidgetFactoryManager(123);
Code is more constrained than prose, but it's also more repetitive, which might allow token sequences to leak into nearby code. It's not a problem for prose, because we don't quote each other with every other sentence.
It's going to arbitrarily choose among identifiers that the LLM thinks are equally good. That will depend on context. For example, the LLM might prefer to maintain consistency with surrounding...
It's going to arbitrarily choose among identifiers that the LLM thinks are equally good. That will depend on context. For example, the LLM might prefer to maintain consistency with surrounding code, which means that in a particular context, a consistent choice is better. Or, maybe there's a style guide in context?
It will also change as models get smarter and/or more opinionated about good coding style.
Yeah, but I'm talking about the case where there's already a choice between equally-good next tokens: Here, "equally likely" is based on all the context you mentioned: surrounding code, style...
Yeah, but I'm talking about the case where there's already a choice between equally-good next tokens:
Maybe the LLM says "Manager" and "Repository" are equally likely
Here, "equally likely" is based on all the context you mentioned: surrounding code, style guides, all of it. And what I'm saying is, if those two tokens are equally likely after taking all that into consideration (which is plausible), then the watermark algorithm has an opportunity to bias towards one of them.
The bias is determined by an RNG, which is seeded with the last several immediately prior tokens (not the LLM context). So if the last several tokens were "AbstractWidgetFactory", those go into the RNG and bias the coin flip. And in my example, "Manager" wins the biased coin flip, so the overall choice is "AbstractWidgetFactoryManager".
And from there, the choice is set in stone. From now on, it's no longer 50/50; "AbstractWidgetFactory" now has a 99.99% chance of being followed by "Manager". So there's no more wiggle room. It doesn't matter if you're a human, an AI, or a space alien: if you want your code to compile, "AbstractWidgetFactory" must be followed by "Manager".
But the watermark detector doesn't have access to the LLM probability distribution; it cannot even in principle recalculate it, because it doesn't know the exact original prompt. The only thing the detector knows is the RNG seed, which comes from hashing the immediately preceding tokens.
So as I understand it, the watermark detector runs a sliding window over its input, calculating the RNG seed at each position. It doesn't know the LLM percentages, but it can look at a single token and tell you if the current RNG "likes" it or not. For human text, the RNG will only like the next token 50% of the time. But for AI text, the RNG will like it slightly more often, say 51% of the time.
So what happens when the detector reads human-written code with that identifier in it?
AbstractWidgetFactoryManager awfm = new AbstractWidgetFactoryManager(123);
It's going to run its sliding window over "AbstractWidgetFactory", seed its RNG with that, and the RNG looks at what immediately follows: "Manager". Oh, the RNG likes that!
Then it advances the sliding window over some other tokens. The RNG likes half of them.
Then the sliding window is "AbstractWidgetFactory" again, the same seed. And what follows? "Manager", of course! The RNG likes that, just the same as it had liked it before, because same seed.
Then it reads some more tokens. Again, the RNG likes half of them on average.
So the detector adds all those "likes" after reading the entire file, and it discovers that the RNG liked 55% of the tokens. So the file as a whole is flagged as written by AI, because the average "like" rate is too high. But AI didn't write it! The AI wrote a fragment of text that was long enough to contain a watermark, and by necessity, the human author was forced to repeat that fragment of text multiple times in their own program.
Yes, I imagine the watermark detection algorithm won't work as well when the text is hightly constrained by context. It will always need a minimum amount of text to work with, and for...
Yes, I imagine the watermark detection algorithm won't work as well when the text is hightly constrained by context. It will always need a minimum amount of text to work with, and for highly-constrained text, it will need more.
But if it doesn't have enough text to pick up a signal, wouldn't it answer "not detected" most of the time? That is, the input will affect the score randomly and the score will be close to what it would be for random noise.
That's what I'm saying about code: it's more repetitive than prose. You often have to quote snippets of existing code verbatim, and you often do this multiple times in the same program. If...
That's what I'm saying about code: it's more repetitive than prose. You often have to quote snippets of existing code verbatim, and you often do this multiple times in the same program.
If "AbstractWidgetFactoryManager" is long enough to contain a watermarked token, typing that into your code once isn't going to trigger the detector—but in a verbose language, you might have to repeat it multiple times. And there might be multiple such identifiers[1], especially if you're interfacing with a library written by AI.
I think it comes down to:
What's the shortest length of quote that could still carry some watermark with it?
How many times do you need to quote the AI before the watermark adds up to become detectable?
[1] Or combinations of identifiers. Boilerplate like new FooBarManager(new BazQuuxConfig()) might need to be repeated verbatim, and the combination could contain a watermark if the AI had a say in naming the config class.
Obviously this is going to be more of an issue in languages with lots of boilerplate, which is why I'm picking on Java. As a contrast, I would think APL code would be very resistant to watermarking.
From the "what you don't know can't hurt you" department: I have a visceral reaction against this. First of all, "has no effect" is not the same thing as "users can't tell". It's like saying the...
From the "what you don't know can't hurt you" department:
This has no practical impact on outputs. Humans cannot tell the difference, at all.
I have a visceral reaction against this. First of all, "has no effect" is not the same thing as "users can't tell". It's like saying the drug trial was a success because users couldn't tell whether they got the placebo. That's cool, but isn't it more important to know if the drug harms its users?
But even if watermarking didn't degrade the LLM's output at all, I would still be deeply bothered by it. It's the same way I feel about printer watermarks. I'm not an activist or a counterfeiter, so why should I care? Because when I pay money for a tool, I expect it to serve me wholeheartedly. And if it goes out of its way to do something not purely in my best interests, that violates my expectation of what a tool ought to be.
AI chat is often using a random number generator to decide what to write. If you're concerned about LLM's not serving you wholeheartedly, maybe you should be concerned about that, too? They're...
AI chat is often using a random number generator to decide what to write. If you're concerned about LLM's not serving you wholeheartedly, maybe you should be concerned about that, too? They're rolling the dice to decide what to tell you! How does that serve you?
Swapping one random number generator for another isn't going to change that.
Though of course it's not just random. The weights bias the results, making some answers much more likely than others.
Overall, the answers being chosen from are in some sense equivalent. Even though they might not seem at all equivalent to you, the AI has no preference between them.
But perhaps a better model would have a preference? That's pretty much what happens when switching to an improved model.
Running the same query multiple times can be a good way of seeing what a model considers to be equivalent. Though, maybe this watermarking scheme would reduce the variety since it's using a biased generator? It's not going to make it better or worse on average, but it will reduce the number of answers that it's choosing from.
Because in an ideal world, the probability distribution of the dice is optimized to serve me. I draw a distinction between the variance versus what causes it. Humans are random, too! If I go to...
They're rolling the dice to decide what to tell you! How does that serve you?
Because in an ideal world, the probability distribution of the dice is optimized to serve me.
I draw a distinction between the variance versus what causes it. Humans are random, too! If I go to the doctor, there's lots of factors I can't control: my name reminds them of a previous case, they didn't sleep well last night, a colleague told them a funny story over lunch, etc… any of which could subtly affect the treatment I receive. That's variance. But it's inherent to the process: if I see my doctor yawn, I'm not gonna go "how dare you get a bad night's sleep" because everyone's day has random events. They might make mistakes, but as long as they're trying their best to uphold their oath, I'm happy.
Now, suppose on my next visit, I see new brochures for a pharmaceutical company. Alongside all the existing variance, there's an additional random event: whether this company wined and dined my doctor. I can't personally tell if it caused my own medical care to change. Maybe they prescribe a certain medication 1% more often, but you'd never know unless you did statistics across thousands of patients. And maybe it's a decent medication—it doesn't always work, but it's about as effective as the physical therapy they would have otherwise prescribed. No net harm.
But I would still feel uneasy. Even though the "before" and "after" are in some sense equivalent, the doctor still allowed an additional influence on their practice, whose purpose was not exclusively to make me healthier.
The probability distribution is the output of the LLM. That's what LLMs are. They take in a context window of tokens, represented as a matrix of CxD, where C is the length of the context window...
Because in an ideal world, the probability distribution of the dice is optimized to serve me.
The probability distribution is the output of the LLM. That's what LLMs are. They take in a context window of tokens, represented as a matrix of CxD, where C is the length of the context window and D is the dimensionality of the internal tokenizer, and output a vector of size 1xD.
That vector represents the probability the next token should be each of the possible tokens. When a LLM then generates text, what it's doing is sampling from that distribution. Changing the PRNG does not change the probability distribution.
Well, that's not the output that users receive. If you want to get technical about it, the output the user receives is a sampling of a distribution. Digging into the paper linked to from the...
The probability distribution is the output of the LLM.
Well, that's not the output that users receive. If you want to get technical about it, the output the user receives is a sampling of a distribution.
Digging into the paper linked to from the article:
Generative watermarking (Fig. 1, bottom) works by carefully modifying the next-token sampling procedure to inject subtle, context-specific modifications into the generated text distribution.
Are they modifying the sampling procedure? The distribution? Some of this is just quibbling over definitions, but fortunately they supply multiple mathematical definitions about the degree to which output texts are distorted:
Choosing the level of non-distortion involves a trade-off; weaker levels of non-distortion can reduce text quality and diversity, whereas stronger levels of non-distortion can reduce detectability and increase computational complexity (Supplementary Information section G.3). For our experiments, we configure SynthID-Text to be single-sequence non-distortionary; this preserves text quality and provides good detectability, while having some reduction to inter-response diversity.
My takeaways from that:
There are different settings. They ran their experiment on "low" and their users didn't notice.
Even on "low", it affected diversity of responses. Which @skybrian kind of mentioned already: you can generate multiple times to get second opinions, but due to the biased sampling you'll get less new information per generation.
It might not be set on "low" for you. Your provider can adjust the dial to whatever feels right to them—which might not be aligned with your preferences.
Your provider might not even be using this algorithm! They might be watermarking using red/green lists, which, as I understand it (I could be wrong), bias generation in a way that is less mathematically ideal than SynthID.
I will say this: if we have to have watermarking, I do believe Scott Aaronson and friends did their homework here, and I'd rather have watermarking that preserves subtle mathematical properties (assuming that vendors have it at the lowest setting). But it's a matter of principle to me. Going back to the doctor analogy, even if you could mathematically prove to me that the pharmaceutical company's visit did not reduce my quality of care, that little doubt is still going to stick around in the back of my mind: did my doctor prescribe this because it's best for me, or for someone else?
Let me try to explain it a slightly different way: Yes, some randomly-chosen word choices are better and others are worse. However, the LLM doesn't "know" that. The AI uses the random number...
Let me try to explain it a slightly different way:
Yes, some randomly-chosen word choices are better and others are worse. However, the LLM doesn't "know" that. The AI uses the random number generator to choose among what the LLM considers to be "equivalent" paraphrases. If they don't seem equivalent to you, it's because you know better than the model.
Sometimes the LLM does "know" that one word is better than other in a given circumstance. For example one word is the right answer and another word is wrong. But if it knew that, it wouldn't defer its choice to the random number generator. The probability distribution would be so skewed that it forces the right answer.
How can an AI lab optimize the probability distribution to serve you better? They could train a new model or continue to train one that they already have. Messing with the random number generator isn't going to do it.
This is counterintuitive because we know that word choices matter when we write, so how can choosing a different word not matter? Surely there are subtle grades of meaning to consider when picking one synonym over another?
That's going to depend on what words come before. In some positions, the next word will be highly constrained so it will always pick the next word the same way. There won't be any randomness at that position. At other places, the next word is a freer choice.
This won't prevent the AI labs from working on improving AI writing style. As training gets better, LLM's will become more sensitive about word choices and less likely to choose randomly. So, the amount of freedom delegated to the random number generator will drop. Probably not to zero, though!
From the article:
[...]
[...]
[...]
I wonder if one reason why people might seem angry with Anthropic's watermarking specifically is that Claude's specific writing style is so idiosyncratic and grating, to the point of being like a human-discernible watermark, and is so difficult to reliably change, that they assume, perhaps even unconsciously, the watermarking must have something to do with it.
What does make me angry is that, the last time I checked, Google's "public detector" for their watermarking is not a public detector, it's a private detector that their public LLM is allowed to use. You are forced to use their LLM chat, hope that their LLM decides to run the detector given your prompt, and then interpret its answer, along with whatever Google-perspective moralizing it added.
smoking gun
blast radius
that confirms it
It's the Mmm Bop of 2026. To the point I will question the sentinence of any human that I hear using them.
It's a real pattern, and you're right to point it out.
The honest assessment is that code is genuinely load-bearing.
Great! I have what I need. Here's the truth...
and on and on and on
I hate these machines and I hate that they wear the faces of humans and I hate that I am required to use them at work :(
I use blast radius frequently when discussing the, well, blast radius of a feature or fix. It's a good term and useful shorthand for the connectivity of a system. I'll stop using it when I'm dead or retired.
It is, but it is somehow the way that Claude weave it into its babble that makes it sound so much worse. Like Mmm Bop 29 years ago. Fine once in awhile. Less fine when it was every 3rd song on the radio for a year.
I think you're right. The claudisms are really annoying and while this watermarking is something different, it's easy to confuse.
I've been using this skill, it's pretty solid: https://github.com/oaustegard/claude-skills/tree/main/declauding
Here are Scott Aaronson's comments.
So if the writing quality isn't that great to begin with, this doesn't make it worse. Any of the samples would use a similar number of AI catch-phrases.
...
I think that might be read in a way that's uncharitable to LLMs and/or the method. Even if writing quality was good. There's still arguably a lot of variation in how any given human could write any given piece of text, depending on minute factors that basically boil down to pseudo randomness. Or in other words, even best-human level performance on the task of writing general text carries enough randomness that a perfect machine replica of it allows for this method at no loss of output quality. Wherever output quality of LLMs is currently lagging to the degree that humans use a lower-entropy next-token distribution (i.e. are more confident about the next word - let's just say in poetry. Also, this is a proper subset of "LLMs lag human performance"), there this watermarking mechanism will stop applying if/when models catch up to humans.
It's really quite ingenious. I lowkey fear what some people will do with the results of this watermark, mostly in over-interpreting that a text is AI to mean that the thought behind it is too. And/or applying the slop label by default, which is also common. But I think it's still a net-positive move.
I don't understand how anyone with a basic understanding of LLMs and/or randomness can believe this would affect output in any way. All they've changed is that the randomness is seeded and they keep the seed to reference later. This has to be a case of people talking about tech while lacking literacy in it, right?
Anyway, this sounds objectively good. I'd prefer if all AI companies spontaneously combusted, but I'm glad this is now a law. But I'm a bit dubitative about how this will actually work. How can it check text against the seed without having the prompt that generated it? If the chatbot's owners are the one providing the checking API, how do I trust that they're not just lying? How much does it actually take for 'human-edited' output to lose enough of its watermark to be indistinguishable from actual human written text? There's a lot of vagueness I'm worried about here.
Anthropic links to this paper. Apparently the tokens are chosen to score highly according to a random scoring function. This is not very clear to me, but here’s the key bit:
Google’s AI summary seems a bit clearer, so maybe the thing to do is to ask your friendly AI what this all means?
It sounds like they start with the scoring functions and produce the text specifically so it scores high. So, they only need the scoring functions to verify the text. But they do need to keep them secret to prevent forgery, so we will have to rely on their API to tell us whether the text was from their LLM or not. Yeah, that’s not ideal.
Likewise, if I suspect a given piece of writing is AI generated, but the checking tools are closed source, does that now mean I need to check the writing against Claude and ChatGPT and Gemini and DeepSeek and etc etc? Each of them are competing, so I don’t see them working together to build a unified “check here” single combined platform unless they’re required by law.
Yeah everything I've seen about this seems like a long winded way of saying "we're using fixed seed(s) for the random number generator"
This was the interesting quote to me, and I do think that makes it clearer what it is doing, rather than calling it a "watermark."
I'm also fairly "AI-luddite" and don't use them hardly at all, aside from boilerplate code (since my work is pushing us to use AI for ~something~), so this change likely wouldn't affect me at all.
Interestingly enough, I saw this after reading 404 Media's Anthropics text watermarking proves ai companies do not care at all about writing, which is a different take on the watermarking, from a perspective I didn't really see echoed in this article.
But I think it's interesting because it seems like 404 media is generally against AI use for creative works, but then is also seemingly against this watermarking that would let it be more detectable?
It's a weird article, and it suggests to me that the author fundamentally misunderstood how LLM technology actually works up until the watermark thing forced him to look into it.
Strangely, he still seems to think that Claude and other LLMs use some completely different, non-LLM method to generate code. It's true that an LLM's code output is going to be more constrained than, say, its poetry output, but that's because LLMs are trained on code examples that are written in a highly constrained way and poetry examples that are written in a far less constrained way — because the level of constraint is a pretty inherent difference between code and poetry.
This means the watermark would be easier to detect for LLM-written poetry than for more highly regimented text, like a legal document where there just isn't much room for randomness. It has little to do with how much AI companies care about generating legalese versus poetry.
If you crank up the randomness on an LLM, you start getting some really weird gibberish that's a mixture of code, creative writing, technical documentation, slang, etc., etc. because the LLM doesn't actually "know" that these are all different kinds of text. They all go into the same training pot.
I did not read the 404 article so I may be misunderstanding you but there seems like there is still a great deal of decision space for an LLM to use to embed a signature while writing code when taking into consideration just things like white space, comments, bracketing styles etc. And then there are lots of different ways to write code that will compile down to the same or very similar machine/byte code. I do wonder how well it will survive e.g. linting but I imagine the intended purpose is mostly for written language rather than code.
According to Anthropic, there will be less watermarking in code, but you're right that it will show up more strongly in places where there's more leeway in which tokens the LLM can choose, like comments:
I imagine that the more strict the syntax has to be (e.g., if a company requires a very uniform code style), that harder it will be to differentiate Claude-generated code from human-written code. Which makes sense, because these constraints also make it harder to tell one human writer from another.
I'm curious how this will affect stuff like class and function names.
A long name like
AbstractWidgetFactoryconsists of multiple tokens, which might be long enough to start to see watermark effects. What do you call the class that coordinates them? Maybe the LLM says "Manager" and "Repository" are equally likely, but given the last few tokens, the watermark algorithm biases towardsAbstractWidgetFactoryManager.Now you've got a watermarked sequence of tokens that you have to repeat verbatim everywhere, regardless of whether you're AI or not:
Code is more constrained than prose, but it's also more repetitive, which might allow token sequences to leak into nearby code. It's not a problem for prose, because we don't quote each other with every other sentence.
It's going to arbitrarily choose among identifiers that the LLM thinks are equally good. That will depend on context. For example, the LLM might prefer to maintain consistency with surrounding code, which means that in a particular context, a consistent choice is better. Or, maybe there's a style guide in context?
It will also change as models get smarter and/or more opinionated about good coding style.
Yeah, but I'm talking about the case where there's already a choice between equally-good next tokens:
Here, "equally likely" is based on all the context you mentioned: surrounding code, style guides, all of it. And what I'm saying is, if those two tokens are equally likely after taking all that into consideration (which is plausible), then the watermark algorithm has an opportunity to bias towards one of them.
The bias is determined by an RNG, which is seeded with the last several immediately prior tokens (not the LLM context). So if the last several tokens were "AbstractWidgetFactory", those go into the RNG and bias the coin flip. And in my example, "Manager" wins the biased coin flip, so the overall choice is "AbstractWidgetFactoryManager".
And from there, the choice is set in stone. From now on, it's no longer 50/50; "AbstractWidgetFactory" now has a 99.99% chance of being followed by "Manager". So there's no more wiggle room. It doesn't matter if you're a human, an AI, or a space alien: if you want your code to compile, "AbstractWidgetFactory" must be followed by "Manager".
But the watermark detector doesn't have access to the LLM probability distribution; it cannot even in principle recalculate it, because it doesn't know the exact original prompt. The only thing the detector knows is the RNG seed, which comes from hashing the immediately preceding tokens.
So as I understand it, the watermark detector runs a sliding window over its input, calculating the RNG seed at each position. It doesn't know the LLM percentages, but it can look at a single token and tell you if the current RNG "likes" it or not. For human text, the RNG will only like the next token 50% of the time. But for AI text, the RNG will like it slightly more often, say 51% of the time.
So what happens when the detector reads human-written code with that identifier in it?
So the detector adds all those "likes" after reading the entire file, and it discovers that the RNG liked 55% of the tokens. So the file as a whole is flagged as written by AI, because the average "like" rate is too high. But AI didn't write it! The AI wrote a fragment of text that was long enough to contain a watermark, and by necessity, the human author was forced to repeat that fragment of text multiple times in their own program.
Yes, I imagine the watermark detection algorithm won't work as well when the text is hightly constrained by context. It will always need a minimum amount of text to work with, and for highly-constrained text, it will need more.
But if it doesn't have enough text to pick up a signal, wouldn't it answer "not detected" most of the time? That is, the input will affect the score randomly and the score will be close to what it would be for random noise.
That's what I'm saying about code: it's more repetitive than prose. You often have to quote snippets of existing code verbatim, and you often do this multiple times in the same program.
If "AbstractWidgetFactoryManager" is long enough to contain a watermarked token, typing that into your code once isn't going to trigger the detector—but in a verbose language, you might have to repeat it multiple times. And there might be multiple such identifiers[1], especially if you're interfacing with a library written by AI.
I think it comes down to:
[1] Or combinations of identifiers. Boilerplate like
new FooBarManager(new BazQuuxConfig())might need to be repeated verbatim, and the combination could contain a watermark if the AI had a say in naming the config class.Obviously this is going to be more of an issue in languages with lots of boilerplate, which is why I'm picking on Java. As a contrast, I would think APL code would be very resistant to watermarking.
From the "what you don't know can't hurt you" department:
I have a visceral reaction against this. First of all, "has no effect" is not the same thing as "users can't tell". It's like saying the drug trial was a success because users couldn't tell whether they got the placebo. That's cool, but isn't it more important to know if the drug harms its users?
But even if watermarking didn't degrade the LLM's output at all, I would still be deeply bothered by it. It's the same way I feel about printer watermarks. I'm not an activist or a counterfeiter, so why should I care? Because when I pay money for a tool, I expect it to serve me wholeheartedly. And if it goes out of its way to do something not purely in my best interests, that violates my expectation of what a tool ought to be.
AI chat is often using a random number generator to decide what to write. If you're concerned about LLM's not serving you wholeheartedly, maybe you should be concerned about that, too? They're rolling the dice to decide what to tell you! How does that serve you?
Swapping one random number generator for another isn't going to change that.
Though of course it's not just random. The weights bias the results, making some answers much more likely than others.
Overall, the answers being chosen from are in some sense equivalent. Even though they might not seem at all equivalent to you, the AI has no preference between them.
But perhaps a better model would have a preference? That's pretty much what happens when switching to an improved model.
Running the same query multiple times can be a good way of seeing what a model considers to be equivalent. Though, maybe this watermarking scheme would reduce the variety since it's using a biased generator? It's not going to make it better or worse on average, but it will reduce the number of answers that it's choosing from.
Because in an ideal world, the probability distribution of the dice is optimized to serve me.
I draw a distinction between the variance versus what causes it. Humans are random, too! If I go to the doctor, there's lots of factors I can't control: my name reminds them of a previous case, they didn't sleep well last night, a colleague told them a funny story over lunch, etc… any of which could subtly affect the treatment I receive. That's variance. But it's inherent to the process: if I see my doctor yawn, I'm not gonna go "how dare you get a bad night's sleep" because everyone's day has random events. They might make mistakes, but as long as they're trying their best to uphold their oath, I'm happy.
Now, suppose on my next visit, I see new brochures for a pharmaceutical company. Alongside all the existing variance, there's an additional random event: whether this company wined and dined my doctor. I can't personally tell if it caused my own medical care to change. Maybe they prescribe a certain medication 1% more often, but you'd never know unless you did statistics across thousands of patients. And maybe it's a decent medication—it doesn't always work, but it's about as effective as the physical therapy they would have otherwise prescribed. No net harm.
But I would still feel uneasy. Even though the "before" and "after" are in some sense equivalent, the doctor still allowed an additional influence on their practice, whose purpose was not exclusively to make me healthier.
The probability distribution is the output of the LLM. That's what LLMs are. They take in a context window of tokens, represented as a matrix of CxD, where C is the length of the context window and D is the dimensionality of the internal tokenizer, and output a vector of size 1xD.
That vector represents the probability the next token should be each of the possible tokens. When a LLM then generates text, what it's doing is sampling from that distribution. Changing the PRNG does not change the probability distribution.
Well, that's not the output that users receive. If you want to get technical about it, the output the user receives is a sampling of a distribution.
Digging into the paper linked to from the article:
Are they modifying the sampling procedure? The distribution? Some of this is just quibbling over definitions, but fortunately they supply multiple mathematical definitions about the degree to which output texts are distorted:
My takeaways from that:
I will say this: if we have to have watermarking, I do believe Scott Aaronson and friends did their homework here, and I'd rather have watermarking that preserves subtle mathematical properties (assuming that vendors have it at the lowest setting). But it's a matter of principle to me. Going back to the doctor analogy, even if you could mathematically prove to me that the pharmaceutical company's visit did not reduce my quality of care, that little doubt is still going to stick around in the back of my mind: did my doctor prescribe this because it's best for me, or for someone else?
Let me try to explain it a slightly different way:
Yes, some randomly-chosen word choices are better and others are worse. However, the LLM doesn't "know" that. The AI uses the random number generator to choose among what the LLM considers to be "equivalent" paraphrases. If they don't seem equivalent to you, it's because you know better than the model.
Sometimes the LLM does "know" that one word is better than other in a given circumstance. For example one word is the right answer and another word is wrong. But if it knew that, it wouldn't defer its choice to the random number generator. The probability distribution would be so skewed that it forces the right answer.
How can an AI lab optimize the probability distribution to serve you better? They could train a new model or continue to train one that they already have. Messing with the random number generator isn't going to do it.