I found the easiest tell to be that the AI writing had no "rough edges". The human writing always had a word or phrase or twist in meaning I wouldn't have expected, whereas the AI writing is...
I found the easiest tell to be that the AI writing had no "rough edges". The human writing always had a word or phrase or twist in meaning I wouldn't have expected, whereas the AI writing is stylistically monotone. That momentary pause where you think to yourself "wait, what was that?" feels important, as that's when you have to shift your perspective to align with the author's; AI writing never asks that of you.
Yes. After I finished and realized I had mostly picked AI writings, I realized it was because I was judging based on what felt more cohesive, instead of what told the better story. The human...
Yes. After I finished and realized I had mostly picked AI writings, I realized it was because I was judging based on what felt more cohesive, instead of what told the better story. The human writings have a momentum, a direction to where they're moving, and re-reading the same paragraph over and over exposes this rawness.
I found it quite easy to distinguish the human ones from AI. The AI ones were bland and felt like reading a corporate pamphlet. It is just beige prose. The human ones, I could feel the personality...
I found it quite easy to distinguish the human ones from AI. The AI ones were bland and felt like reading a corporate pamphlet. It is just beige prose. The human ones, I could feel the personality of the writer while reading it. Even in the Sagan one which is kind of mundane, the choice of imagery of the penultimate phrase was a dead giveaway that it was human.
Same. Went 5 for 5 (though went back and forth on the Carl Sagan one), and thought it was easy. I also think that this A/B test was helped by the fact that the text was so short. This allows the...
Same. Went 5 for 5 (though went back and forth on the Carl Sagan one), and thought it was easy.
I also think that this A/B test was helped by the fact that the text was so short. This allows the AI to leverage the fact that the actual prompt was likely doing a lot of the work in imbuing character here.
So, I run a literary magazine. (Actually, a couple.) But in the last 9 months or so, the amount of LLM generated writing that's been submitted has been appalling. It's so, so frustrating to me,...
So, I run a literary magazine. (Actually, a couple.) But in the last 9 months or so, the amount of LLM generated writing that's been submitted has been appalling. It's so, so frustrating to me, because I can never be sure unless the submitter leaves in their prompts (which absolutely has happened and continues to happen), and I want to give strangers the benefit of the doubt, but there are particular structures that LLMs just LOVE to follow. Lists of three, which I think everyone knows by now, but it also just loves to do this:
[Thing]—not [this thing], but [this other thing]. "Not, but" seems to be its favorite rhetorical move to try to create some kind of comparison. It shows up in the example that @stu2b50 included in their message, even. It's like nails on a chalkboard to me now.
It really does make me roll my eyes to see the proliferation of LLM writing. Even when there might be valuable information in some of educational content put out, it makes me feel so bored to read...
It really does make me roll my eyes to see the proliferation of LLM writing. Even when there might be valuable information in some of educational content put out, it makes me feel so bored to read through them.
Different LLMs have slightly different writing styles but in the end, whatever prompt results come out sounds more like them. And not to mention, the more you read something, the more it influences your own language choices. So, even unintentionally, if you're spending time on the internet and it becomes a majority of AI generated text, you will also write like AI.
I used to be very particular about grammar and now, I lean harder into having a distinct writing style over technically correct sentences, even in formal communication. Also, being more intentional about what I'm reading and refining my writing style in general just so it can sound more interesting.
I'm super curious about literary magazines and the trends in the kind of writing you're seeing beyond LLMs, can I reach out to you to talk more about it?
I’ve used LLM’s often for bullshit but I’ve gotten used to the way it writes and the metaphors and similes it likes to use. It’s repetitive and seems to be consistent among different prompts. It...
I’ve used LLM’s often for bullshit but I’ve gotten used to the way it writes and the metaphors and similes it likes to use. It’s repetitive and seems to be consistent among different prompts. It actually takes a bit of work to get it to sound somewhat natural which at that point you might as well write the thing yourself.
Some of the NYT comments pointed at why this is a mostly unhelpful quiz (shock!) Stuff like, most of the human writing was not contemporary/21st century, two sentences is too small to pick from,...
Some of the NYT comments pointed at why this is a mostly unhelpful quiz (shock!) Stuff like, most of the human writing was not contemporary/21st century, two sentences is too small to pick from, and there needed to be an “I don’t like either” option.
That said, I imagine AI writing will continue improve and eventually be indistinguishable or preferred for a lot of people (myself included, as I picked AI 3/5 times in the quiz.
As always, I think of Jeff Goldblum in Jurrasic Park saying the scientists were so preoccupied with if they could to never stop and think about if they should.
I think the framing effect is fascinating. Basically everyone who has commented thus far appears to have sidestepped the actual question and replaced it with a task of identifying AI written text....
I think the framing effect is fascinating. Basically everyone who has commented thus far appears to have sidestepped the actual question and replaced it with a task of identifying AI written text. The unstated rule being that human text is definitionally enjoyable, and enjoying AI text is therefore an error.
For my part, I identified 4/5 AI texts, and preferred AI in 2/5 (science and historical fiction).
Pretty fun quiz. Surprisingly hard to distinguish. I did manage to get human for all of them but 1 (the Carl Sagan one). The average quiz-taker, as of now, evidently prefers AI over humans in...
Pretty fun quiz. Surprisingly hard to distinguish. I did manage to get human for all of them but 1 (the Carl Sagan one).
The average quiz-taker, as of now, evidently prefers AI over humans in writing quality - that's already a heavily biased sample towards book-readers, so I can only imagine if you gave it to the "average" American.
Spoilers
I really had to read them closely; the main thing is that the AI passages can be a bit plain or underwhelming in underlying message. Ignoring the first one, since Cormack has such a distinctive writing style in the first question,
The healers teach that every remedy extracts its cost. A fever brought down will rise again somewhere; a wound closed by magic leaves its scar on the world, invisible but present. This is why the wise hesitate. Not from cruelty, but from understanding that interference ripples outward in ways we cannot trace. To cure a blight may curse a harvest three valleys over. Power is not the difficult thing. Restraint is the difficult thing.
This is like a pretty basic theme, he ho unintentional consequences yadada.
That being said, that's like a pretty close reading - if you plopped one of these paragraphs as just one in thousands, it'd be pretty hard, for me, at least to tell if I only read it once.
That's what got me on the Carl Sagan one, felt kinda basic.
Seems like most people prefer AI writing and that’s even from people who use the NYTimes a lot which is worrying. The only AI one I chose was the science based one.
Seems like most people prefer AI writing and that’s even from people who use the NYTimes a lot which is worrying. The only AI one I chose was the science based one.
I think this is an unfair comparison. These are all isolated passages from books. They have stripped the entire mental model that a reader has built up before reaching them. So it is silly to...
I think this is an unfair comparison. These are all isolated passages from books. They have stripped the entire mental model that a reader has built up before reaching them. So it is silly to compare their comprehensibility to passages which I imagine were generated with a prompt meant to optimize it.
Plus, (someone on bsky pointed this out) if you Google many of those AI written phrases you'll find that humans also wrote them. The AI is only writing what humans already created. Plagiarism...
Plus, (someone on bsky pointed this out) if you Google many of those AI written phrases you'll find that humans also wrote them. The AI is only writing what humans already created.
Precisely. A paragraph can never be "AI-written" because an LLM is incapable of writing. It merely generates output based on input. And that generation is built upon a foundation of human creation.
Precisely. A paragraph can never be "AI-written" because an LLM is incapable of writing. It merely generates output based on input. And that generation is built upon a foundation of human creation.
I found the easiest tell to be that the AI writing had no "rough edges". The human writing always had a word or phrase or twist in meaning I wouldn't have expected, whereas the AI writing is stylistically monotone. That momentary pause where you think to yourself "wait, what was that?" feels important, as that's when you have to shift your perspective to align with the author's; AI writing never asks that of you.
Yes. After I finished and realized I had mostly picked AI writings, I realized it was because I was judging based on what felt more cohesive, instead of what told the better story. The human writings have a momentum, a direction to where they're moving, and re-reading the same paragraph over and over exposes this rawness.
I found it quite easy to distinguish the human ones from AI. The AI ones were bland and felt like reading a corporate pamphlet. It is just beige prose. The human ones, I could feel the personality of the writer while reading it. Even in the Sagan one which is kind of mundane, the choice of imagery of the penultimate phrase was a dead giveaway that it was human.
Same. Went 5 for 5 (though went back and forth on the Carl Sagan one), and thought it was easy.
I also think that this A/B test was helped by the fact that the text was so short. This allows the AI to leverage the fact that the actual prompt was likely doing a lot of the work in imbuing character here.
So, I run a literary magazine. (Actually, a couple.) But in the last 9 months or so, the amount of LLM generated writing that's been submitted has been appalling. It's so, so frustrating to me, because I can never be sure unless the submitter leaves in their prompts (which absolutely has happened and continues to happen), and I want to give strangers the benefit of the doubt, but there are particular structures that LLMs just LOVE to follow. Lists of three, which I think everyone knows by now, but it also just loves to do this:
[Thing]—not [this thing], but [this other thing]. "Not, but" seems to be its favorite rhetorical move to try to create some kind of comparison. It shows up in the example that @stu2b50 included in their message, even. It's like nails on a chalkboard to me now.
It really does make me roll my eyes to see the proliferation of LLM writing. Even when there might be valuable information in some of educational content put out, it makes me feel so bored to read through them.
Different LLMs have slightly different writing styles but in the end, whatever prompt results come out sounds more like them. And not to mention, the more you read something, the more it influences your own language choices. So, even unintentionally, if you're spending time on the internet and it becomes a majority of AI generated text, you will also write like AI.
I used to be very particular about grammar and now, I lean harder into having a distinct writing style over technically correct sentences, even in formal communication. Also, being more intentional about what I'm reading and refining my writing style in general just so it can sound more interesting.
I'm super curious about literary magazines and the trends in the kind of writing you're seeing beyond LLMs, can I reach out to you to talk more about it?
Sure, always happy to talk more about literary mags!
I’ve used LLM’s often for bullshit but I’ve gotten used to the way it writes and the metaphors and similes it likes to use. It’s repetitive and seems to be consistent among different prompts. It actually takes a bit of work to get it to sound somewhat natural which at that point you might as well write the thing yourself.
Some of the NYT comments pointed at why this is a mostly unhelpful quiz (shock!) Stuff like, most of the human writing was not contemporary/21st century, two sentences is too small to pick from, and there needed to be an “I don’t like either” option.
That said, I imagine AI writing will continue improve and eventually be indistinguishable or preferred for a lot of people (myself included, as I picked AI 3/5 times in the quiz.
As always, I think of Jeff Goldblum in Jurrasic Park saying the scientists were so preoccupied with if they could to never stop and think about if they should.
I think the framing effect is fascinating. Basically everyone who has commented thus far appears to have sidestepped the actual question and replaced it with a task of identifying AI written text. The unstated rule being that human text is definitionally enjoyable, and enjoying AI text is therefore an error.
For my part, I identified 4/5 AI texts, and preferred AI in 2/5 (science and historical fiction).
Pretty fun quiz. Surprisingly hard to distinguish. I did manage to get human for all of them but 1 (the Carl Sagan one).
The average quiz-taker, as of now, evidently prefers AI over humans in writing quality - that's already a heavily biased sample towards book-readers, so I can only imagine if you gave it to the "average" American.
Spoilers
I really had to read them closely; the main thing is that the AI passages can be a bit plain or underwhelming in underlying message. Ignoring the first one, since Cormack has such a distinctive writing style in the first question,
This is like a pretty basic theme, he ho unintentional consequences yadada.
That being said, that's like a pretty close reading - if you plopped one of these paragraphs as just one in thousands, it'd be pretty hard, for me, at least to tell if I only read it once.
That's what got me on the Carl Sagan one, felt kinda basic.
Seems like most people prefer AI writing and that’s even from people who use the NYTimes a lot which is worrying. The only AI one I chose was the science based one.
I think this is an unfair comparison. These are all isolated passages from books. They have stripped the entire mental model that a reader has built up before reaching them. So it is silly to compare their comprehensibility to passages which I imagine were generated with a prompt meant to optimize it.
Plus, (someone on bsky pointed this out) if you Google many of those AI written phrases you'll find that humans also wrote them. The AI is only writing what humans already created.
Plagiarism machine plagiarizes and all that.
Precisely. A paragraph can never be "AI-written" because an LLM is incapable of writing. It merely generates output based on input. And that generation is built upon a foundation of human creation.
Is this not a distinction without a difference? Do humans not generate output based on input or recombine others' words in new ways?
No, I think humans make things, AIs are not people and the people-ness in intentionally choosing words matters.
I don't read enough
For the life of me I can't understand what this means. How can I choose the writing I like better regardless of how it's written?
They mean 'regardless of whether a human or ai wrote it,' but I agree it's super unclear.
I think they’re trying to say to just go with your gut and not try to analyze if it’s “correctly” written or try to sus out which is the AI one.