This is a solid essay, the author frames post training/fine tuning accurately, which is somewhat rare at the moment. Generally I agree with all the parts, but I don't agree about what they add up...
This is a solid essay, the author frames post training/fine tuning accurately, which is somewhat rare at the moment. Generally I agree with all the parts, but I don't agree about what they add up to.
It's true that pangram and grammerly are imperfect, the latter especially. It doesn't make sense to trust their output. Pangram is good enough to be considered a signal, but it's not good enough to be considered definitive proof.
It's likely true that post training favors negative parallelism because that language structure is an artifact of training models to "reason". That's the most interesting part of the essay IMO. The models achieve their ability to mimic reasoning through language, so innocuous phrases like "but wait" and "I'm reconsidering" have functional value to LLMs far in excess of their value in language from a human perspective. For an LLM they're like the equivalent of cognitive processes.
Negative parallelism: "It's not X, it's Y" is potentially a representation of "most obvious inference" but instead "somewhat less obvious but possibly more correct inference". And it's true that similar reasoning is useful for people.
But I don't think this follows:
In the end, shaming people for writing that gets flagged as AI can lead people to sidestep structures the model has learned from us: structures that are effective tools for argumentation. We take the tools of critical thinking out of the kit at the time we most need them.
I think this encapsulates the logical leap of faith you need to make to get from the premise to the conclusion that we shouldn't shame people for using AI to write. I'd be more likely to agree if AI writing was closer to human writing, but if that were the case it would be almost impossible to detect and there'd be nothing to talk about.
The models did learn these things from our language, so it's fair to say "from us", but then after that they went through vast amounts of fine tuning that morph what they learned from us into something substantively different. In part because, while our reasoning is greatly influenced by language, it doesn't happen solely through language. We have dedicated circuits for it, so to speak.
The idea that avoiding LLMisms will cause people to lose important reasoning tools is silly IMO. The current generation of LLMisms doesn't cut into the toolbox in any meaningful way. I think that's pretty seriously mischaracterizing human cognition and exaggerating the scope of LLMisms.
Post training causes LLMs to grossly overuse a limited list of rhetorical tricks in obnoxious ways. If that causes people to avoid being similarly obnoxious I think that's a good thing!
I hope people continue to call out LLM writing when authors attempt to pass it off as their own work. That's how a culture has the conversation about what we're going to accept and what we're going to reject. I think that's an important conversation where LLM writing is concerned.
That said, witch hunts are no good and it's not always obvious whether or not something is LLM written. Em-dashes are the most famous example of this. For a while now they've been almost guaranteed to draw AI accusations on social media even though, used judiciously, they're perfectly reasonable punctuation that humans have been using forever. Just not anywhere near as much as LLMs use them. Probably in part because they don't show up overtly in most keyboard layouts.
I agree with the spirit of the essay in the sense that we should be mindful about how we approach the issue, rather than reflexively calling things AI or trusting tools like pangram. However, the larger part of LLM generated writing is obvious for all sorts of reasons beyond em-dashes and negative parallelism, not so much because of specific tells, but because —negative parallelism!— of the volume and consistency in a given piece.
I think we'll get better over time at telling the difference between human and LLM writing as we consume more slop because the patterns will become intuitive. We're all carrying around an exhaustive list of language rules and structure we couldn't fully articulate and often don't even know we're aware of. Learning to identify (current) LLM writing isn't a difficult task for our language centers to manage.
I think this is right on the money. I do think you might be underestimating the degree to which using language to say something like "but wait" to oneself is an important cognitive technology for...
The idea that avoiding LLMisms will cause people to lose important reasoning tools is silly IMO. The current generation of LLMisms doesn't cut into the toolbox in any meaningful way. I think that's pretty seriously mischaracterizing human cognition and exaggerating the scope of LLMisms.
Post training causes LLMs to grossly overuse a limited list of rhetorical tricks in obnoxious ways. If that causes people to avoid being similarly obnoxious I think that's a good thing!
I think this is right on the money.
I do think you might be underestimating the degree to which using language to say something like "but wait" to oneself is an important cognitive technology for humans. Without it one can only entertain an inarticulate sense that something isn't right; with it one can diagnose the problem, and consider in turn whether the idea that there's a problem itself has problems.
But The Problem with AI writing isn't that it uses and then somehow discredits these constructions; it's that is uses them inappropriately, and poorly, and generally makes a hash of it in a variety of ways when prompted to produce prose. As you note, AI writing is often obvious for all sorts of reasons. As discussed previously, one of those reasons tends to be that the net result is soulless bullshit that communicates nothing.
Honestly I don't, fundamentally, care if people are using a process other than thinking of each word and then typing it out to put words on the page. But I do care if those words mean nothing, or somehow less than nothing, when I read them. Nobody looks at "the kind of walking that made benches become men" when they pull it from a hat and says "ah yes, that is exactly capturing the idea I need to convey in my story". It's not a thing. It's a previously unexplored area of text space that turns out to be a complete wasteland. We have mapped the semantic void and found out that it sucks there and that we want to go home.
When there are the supposed "tells" of AI writing in text like this, they are being used poorly to convey nothing of value, and those tells are all that is left to see. You cannot hang an entire text worth reading on em-dashes and negative parallelism alone, but boy will AI try.
Good points. I agree that language plays an important part in cognition. Though in my experience a lot of reasoning also happens below language. Or maybe outside of language, definitely not...
Good points. I agree that language plays an important part in cognition. Though in my experience a lot of reasoning also happens below language. Or maybe outside of language, definitely not language mediated.
About the semantic void bit... Despite the fact that I cringe a little when I see LessWrong at the top of the page, I read it anyway. It was really interesting, thanks for the link. It sounds like it was (roughly) about injecting IDs for tokens that don't exist (they're in the void). Essentially accessing inference from weights the normal input path wouldn't ever have access to.
So the inference you get out of LLMs is actually in the opposite space to the semantic void. But correct me if I'm wrong I half skimmed. I thought the implication that the geometric center of the space is void rather than densely populated was compelling, even if it doesn't have any practical implications.
In any case reading LLM generated prose does often feel void-like.
I think I agree with the broad thrust of what you’re saying, but I also think there’s maybe a bit of an overestimate there on how much “LLM writing” is identifiable as a broad category. In my...
I think I agree with the broad thrust of what you’re saying, but I also think there’s maybe a bit of an overestimate there on how much “LLM writing” is identifiable as a broad category. In my experience “ChatGPT writing” or “Claude writing” or whatever else are identifiable as examples of LLM writing, but the question I’d ask is how much that’s a technical consequence of current LLM capabilities and how much it’s a result of millions of people using the same baseline tool with subtle patterns that’ll show through as a result?
Most of the LLM output we’re seeing is the equivalent of McDonald’s food - cheap, mass produced, relatively low effort, identifiable not because it’s noticeably terrible in and of itself but because it’s the exact same not-great-but-fine product we’ve experienced many times before.
I think it’s maybe a bit of a trap to think we can identify LLM writing in general (and in turn perhaps let our guard down a little when we assume something is human written) just because we can identify writing from the most prevalent publicly available LLMs.
Sort of an aside, I kind of wish people would design proofreading tools for people who actually know how to write rather than for people who read/write at an 8th grade level. I actually(!) know...
It's true that pangram and grammerly are imperfect, the latter especially. It doesn't make sense to trust their output. Pangram is good enough to be considered a signal, but it's not good enough to be considered definitive proof.
Sort of an aside, I kind of wish people would design proofreading tools for people who actually know how to write rather than for people who read/write at an 8th grade level. I actually(!) know one of my chronic weaknesses as a writer is a tendency to lean too heavily on pointless qualifiers like “actually,” “just,” “simply,” etc. I write down “just” and “actually” way too much when I write stream-of-consciousness and I really (GAH! I did it again!) only notice it when I go back and see my post later. In the moment I don’t even notice it, it’s like a tic.
I’d love if I could have a simple model with proofreading parameters that I could define for myself running in the background to catch this sort of thing. When I proofread for myself I always catch it and edit it, but if I’m rattling off a post on Tildes or something I will rarely bother to do that extra step. This is a general tendency I notice with how technology is pitched. The narrative is that I am ignorant or insecure or whatever and the enlightened technology will bring the vaster intelligence of crowds and experts to make self-optimize. Fuck that! I know my own voice and how I want to write. I have bad habits, I don’t need a virtual ‘coach’ to tell me how my voice ought to be, I need to break the habits that I’ve identified for myself.
I can see this being especially useful for professional writers who work with professional editors. When you have a close working relationship with someone you do start to learn their tics as well as which edits are uncontroversial habits they have and don’t want. You can literally program a customized proofreader to pre-proof their manuscript before they send it to you so you can speed up the whole review process.
It does feel very much like personalised learning writing agents will be embedded in our devices in the future. Kind of like how spell-checking dictionaries were one-size-fits-all in the...
It does feel very much like personalised learning writing agents will be embedded in our devices in the future. Kind of like how spell-checking dictionaries were one-size-fits-all in the beginning, but now can be configured to "learn" (just adding in acceptable "mistakes", whether slang, profanity, or in-joke misspellings, etc. to a custom dictionary) to fit how I like to write. I would like that. I don't want every phrase I write being uploaded to some central computer and run through that for improvement, though. It runs on my device or I stick with the dumb spellcheck, personally.
Yeah, I can see this being especially useful as a “teaching aid” module in an IED to kind of teach you to code instead of vibing it out for you. Of course it could probably vibe it out for you as...
Yeah, I can see this being especially useful as a “teaching aid” module in an IED to kind of teach you to code instead of vibing it out for you. Of course it could probably vibe it out for you as well, but I expect an ideal agentic coding system would need to operate on some kind of literate programming paradigm where you’re commenting in explanations in plain language of what each section of code does alongside the code itself. Given the costs of using LLM tokens, you probably want your interface conventions to steer people towards understand when and why they’ll want to burn tokens on writing something rather than doing it themselves.
If the LLM’s remain the same, their tells will become more and more recognizable. But I think this is more likely to be a phase? I don’t think Claude says “you’re absolutely right” anymore?
If the LLM’s remain the same, their tells will become more and more recognizable. But I think this is more likely to be a phase? I don’t think Claude says “you’re absolutely right” anymore?
Yep. For reasoning, I don’t particularly mind since polished prose isn’t really the point, but it would be nice if they did a better job on documentation.
Yep. For reasoning, I don’t particularly mind since polished prose isn’t really the point, but it would be nice if they did a better job on documentation.
Recent overuse by language models has led many to declare it bad writing. I'm not so sure. Nobody called JFK a lazy writer when he said, "ask not what your country can do for you – ask what you can do for your country." Negative parallelism is a rhetorical device, and any rhetorical device is only as lazy or inspired as what it contains.
[...]
Now, we have AI detectors that claim to protect you from the witch hunt by looking for these patterns. You take your own writing and you run it through Grammarly, which will analyze word patterns that AI detectors might flag. Then it offers ideas for how to change them, which a) gives Grammarly the power to write for you and b) makes your writing lose any sense of rhythm or intent.
[...]
Defining reasoning the way it has been used in LLMs assumes that the point of asking a question is to get an answer, that answers can be verified, and that nothing is lost in immediate closure. This has real effects on writing, and the openness to doubt is something we lose in the rapid prototyping of thought that occurs with a language model. Ambiguity, doubt, and uncertainty matter more to some ways of thinking than any immediate answer. The inner life grows in the spaces between the industrial complexes that harness every remnant of our externalized thought.
Nonetheless, the language we use in these states is the same. When AI detectors flag text as AI-generated, is it because it follows a certain structural pattern of that reasoning? Pangram and reasoning models both detect structural patterns based on how humans reason when writing. Pangram's model is trained on pre-2021 data; it then inserts AI-generated versions of the same text into its training.
So, if we publicly shame people whose text looks like it might have been written by a machine – because it mimics the language used for human reasoning – and people stop writing in ways that they internalize as "AI writing" out of fear of false detection, it sends a signal that your language for reasoning must be policed, or you too could be held up to public scrutiny.
In the end, shaming people for writing that gets flagged as AI can lead people to sidestep structures the model has learned from us: structures that are effective tools for argumentation. We take the tools of critical thinking out of the kit at the time we most need them.
[...]
I'm not convinced by the old "if you haven't done anything wrong, you don't have anything to worry about" line. I've seen 99.8% cited as a measure of accuracy in automated surveillance systems since 2018. As Arvind Narayanan has noted, that is on a per-paper basis, which compounds every time we use it. So up to 10% of college students could be falsely accused. If we collectively run every bit of text through an AI model to check whether it is AI-generated, we will generate false positives on an even larger scale.
[...]
We create a culture of self-censorship and AI-detector-pressured rewriting and paraphrasing as people strive to avoid these witch hunts. That is the opposite of protecting human expression. We should resist normalizing a trust in any machine's ability to determine matters of guilt. If using AI to write is, at its worst, an industrialization of the mind, then AI detection, at its worst, becomes a surveillance system for thought.
This is a solid essay, the author frames post training/fine tuning accurately, which is somewhat rare at the moment. Generally I agree with all the parts, but I don't agree about what they add up to.
It's true that pangram and grammerly are imperfect, the latter especially. It doesn't make sense to trust their output. Pangram is good enough to be considered a signal, but it's not good enough to be considered definitive proof.
It's likely true that post training favors negative parallelism because that language structure is an artifact of training models to "reason". That's the most interesting part of the essay IMO. The models achieve their ability to mimic reasoning through language, so innocuous phrases like "but wait" and "I'm reconsidering" have functional value to LLMs far in excess of their value in language from a human perspective. For an LLM they're like the equivalent of cognitive processes.
Negative parallelism: "It's not X, it's Y" is potentially a representation of "most obvious inference" but instead "somewhat less obvious but possibly more correct inference". And it's true that similar reasoning is useful for people.
But I don't think this follows:
I think this encapsulates the logical leap of faith you need to make to get from the premise to the conclusion that we shouldn't shame people for using AI to write. I'd be more likely to agree if AI writing was closer to human writing, but if that were the case it would be almost impossible to detect and there'd be nothing to talk about.
The models did learn these things from our language, so it's fair to say "from us", but then after that they went through vast amounts of fine tuning that morph what they learned from us into something substantively different. In part because, while our reasoning is greatly influenced by language, it doesn't happen solely through language. We have dedicated circuits for it, so to speak.
The idea that avoiding LLMisms will cause people to lose important reasoning tools is silly IMO. The current generation of LLMisms doesn't cut into the toolbox in any meaningful way. I think that's pretty seriously mischaracterizing human cognition and exaggerating the scope of LLMisms.
Post training causes LLMs to grossly overuse a limited list of rhetorical tricks in obnoxious ways. If that causes people to avoid being similarly obnoxious I think that's a good thing!
I hope people continue to call out LLM writing when authors attempt to pass it off as their own work. That's how a culture has the conversation about what we're going to accept and what we're going to reject. I think that's an important conversation where LLM writing is concerned.
That said, witch hunts are no good and it's not always obvious whether or not something is LLM written. Em-dashes are the most famous example of this. For a while now they've been almost guaranteed to draw AI accusations on social media even though, used judiciously, they're perfectly reasonable punctuation that humans have been using forever. Just not anywhere near as much as LLMs use them. Probably in part because they don't show up overtly in most keyboard layouts.
I agree with the spirit of the essay in the sense that we should be mindful about how we approach the issue, rather than reflexively calling things AI or trusting tools like pangram. However, the larger part of LLM generated writing is obvious for all sorts of reasons beyond em-dashes and negative parallelism, not so much because of specific tells, but because —negative parallelism!— of the volume and consistency in a given piece.
I think we'll get better over time at telling the difference between human and LLM writing as we consume more slop because the patterns will become intuitive. We're all carrying around an exhaustive list of language rules and structure we couldn't fully articulate and often don't even know we're aware of. Learning to identify (current) LLM writing isn't a difficult task for our language centers to manage.
I think this is right on the money.
I do think you might be underestimating the degree to which using language to say something like "but wait" to oneself is an important cognitive technology for humans. Without it one can only entertain an inarticulate sense that something isn't right; with it one can diagnose the problem, and consider in turn whether the idea that there's a problem itself has problems.
But The Problem with AI writing isn't that it uses and then somehow discredits these constructions; it's that is uses them inappropriately, and poorly, and generally makes a hash of it in a variety of ways when prompted to produce prose. As you note, AI writing is often obvious for all sorts of reasons. As discussed previously, one of those reasons tends to be that the net result is soulless bullshit that communicates nothing.
Honestly I don't, fundamentally, care if people are using a process other than thinking of each word and then typing it out to put words on the page. But I do care if those words mean nothing, or somehow less than nothing, when I read them. Nobody looks at "the kind of walking that made benches become men" when they pull it from a hat and says "ah yes, that is exactly capturing the idea I need to convey in my story". It's not a thing. It's a previously unexplored area of text space that turns out to be a complete wasteland. We have mapped the semantic void and found out that it sucks there and that we want to go home.
When there are the supposed "tells" of AI writing in text like this, they are being used poorly to convey nothing of value, and those tells are all that is left to see. You cannot hang an entire text worth reading on em-dashes and negative parallelism alone, but boy will AI try.
Good points. I agree that language plays an important part in cognition. Though in my experience a lot of reasoning also happens below language. Or maybe outside of language, definitely not language mediated.
About the semantic void bit... Despite the fact that I cringe a little when I see LessWrong at the top of the page, I read it anyway. It was really interesting, thanks for the link. It sounds like it was (roughly) about injecting IDs for tokens that don't exist (they're in the void). Essentially accessing inference from weights the normal input path wouldn't ever have access to.
So the inference you get out of LLMs is actually in the opposite space to the semantic void. But correct me if I'm wrong I half skimmed. I thought the implication that the geometric center of the space is void rather than densely populated was compelling, even if it doesn't have any practical implications.
In any case reading LLM generated prose does often feel void-like.
I think I agree with the broad thrust of what you’re saying, but I also think there’s maybe a bit of an overestimate there on how much “LLM writing” is identifiable as a broad category. In my experience “ChatGPT writing” or “Claude writing” or whatever else are identifiable as examples of LLM writing, but the question I’d ask is how much that’s a technical consequence of current LLM capabilities and how much it’s a result of millions of people using the same baseline tool with subtle patterns that’ll show through as a result?
Most of the LLM output we’re seeing is the equivalent of McDonald’s food - cheap, mass produced, relatively low effort, identifiable not because it’s noticeably terrible in and of itself but because it’s the exact same not-great-but-fine product we’ve experienced many times before.
I think it’s maybe a bit of a trap to think we can identify LLM writing in general (and in turn perhaps let our guard down a little when we assume something is human written) just because we can identify writing from the most prevalent publicly available LLMs.
Sort of an aside, I kind of wish people would design proofreading tools for people who actually know how to write rather than for people who read/write at an 8th grade level. I actually(!) know one of my chronic weaknesses as a writer is a tendency to lean too heavily on pointless qualifiers like “actually,” “just,” “simply,” etc. I write down “just” and “actually” way too much when I write stream-of-consciousness and I really (GAH! I did it again!) only notice it when I go back and see my post later. In the moment I don’t even notice it, it’s like a tic.
I’d love if I could have a simple model with proofreading parameters that I could define for myself running in the background to catch this sort of thing. When I proofread for myself I always catch it and edit it, but if I’m rattling off a post on Tildes or something I will rarely bother to do that extra step. This is a general tendency I notice with how technology is pitched. The narrative is that I am ignorant or insecure or whatever and the enlightened technology will bring the vaster intelligence of crowds and experts to make self-optimize. Fuck that! I know my own voice and how I want to write. I have bad habits, I don’t need a virtual ‘coach’ to tell me how my voice ought to be, I need to break the habits that I’ve identified for myself.
I can see this being especially useful for professional writers who work with professional editors. When you have a close working relationship with someone you do start to learn their tics as well as which edits are uncontroversial habits they have and don’t want. You can literally program a customized proofreader to pre-proof their manuscript before they send it to you so you can speed up the whole review process.
It does feel very much like personalised learning writing agents will be embedded in our devices in the future. Kind of like how spell-checking dictionaries were one-size-fits-all in the beginning, but now can be configured to "learn" (just adding in acceptable "mistakes", whether slang, profanity, or in-joke misspellings, etc. to a custom dictionary) to fit how I like to write. I would like that. I don't want every phrase I write being uploaded to some central computer and run through that for improvement, though. It runs on my device or I stick with the dumb spellcheck, personally.
Yeah, I can see this being especially useful as a “teaching aid” module in an IED to kind of teach you to code instead of vibing it out for you. Of course it could probably vibe it out for you as well, but I expect an ideal agentic coding system would need to operate on some kind of literate programming paradigm where you’re commenting in explanations in plain language of what each section of code does alongside the code itself. Given the costs of using LLM tokens, you probably want your interface conventions to steer people towards understand when and why they’ll want to burn tokens on writing something rather than doing it themselves.
If the LLM’s remain the same, their tells will become more and more recognizable. But I think this is more likely to be a phase? I don’t think Claude says “you’re absolutely right” anymore?
The things I've noticed a lot in its "reasoning" portion: "Let me reconsider" and "Now I've got a clear picture".
Yep. For reasoning, I don’t particularly mind since polished prose isn’t really the point, but it would be nice if they did a better job on documentation.
From the article:
[...]
[...]
[...]
[...]