54
votes
Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task
Link information
This data is scraped automatically and may be incorrect.
- Title
- Project Overview ‹ Your Brain on ChatGPT - MIT Media Lab
- Word count
- 875 words
I thought these notes at the end were particularly amusing, and also kind of important.
The study is currently still in preprint, I believe.
Abstract via ArXiv:
Pretty damning if you ask me, especially that the effects will last for that long.
It’s kind of scary to think what might happen on a societal level if kids of all ages end up becoming overly reliant on these tools in school already and “learn”/expect to outsource their cognitive abilities for life.
What I am wondering is if they also differentiated in how people use LLMs. Because, in my view, there are two very distinct ways of using them for essays and a variety of other tasks.
In my personal, so very much anecdotal, experience the last method still requires people to be very much engaged. But, I think most people who start using LLMs sort of drift to the lazy approach very easily. Because it is very convenient and easy to slowly hand over more thinking to an LLM.
Speaking personally again I now see them as valuable tools in arsenal (I have written about it more in the past), but it has required me to very explicitly think about how I use them, where and why.
I have the same “suspicion” indeed.
For the schooling/learning context especially, having a personalized, (nearly) all-knowing, and always-patient affordable teacher is the eager and engaged student’s dream come true… Whereas for the lazy student, it’s a valuable tool to cheat yourself out of the effort that is learning.
Essentially, these tools could very well within the same classroom both push to even further ceilings and lower lows, which makes this so particularly tricky from a regulatory perspective.
That’s a fascinating idea, since I happened to recently come across a high school teacher saying that since covid, the gap between the highest and lowest performers has widened, and it’s making it very difficult to design or adapt any curriculum that can work for everyone in the classroom! Certainly COVID was a disruption, but I wonder if this double-edged sword of LLMs is also contributing?
That's gotta be mostly a lowered floor. And LLMs contribute to the lowering of the floor. I think a lot of people experienced a combination of cabin fever, proximity to abusers, existential fears, and accumulated a learned helplessness. We also had the curtains around our society pulled back a bit. Apparently we can afford to improve unemployment checks. Kids can be fed 3 meals a day from their school. And most white collar jobs can be done from home with few issues. But then we took it all away once it wasn't the last line of defense against collapse. So now a lot of people, including teachers, school admin, and parents, are left to realize how unfair everything is on top of getting traumatized.
A vaguely similar thing happened to me after I got a high paying job that I'd been trying to earn for decades and then was told by my boss that I didn't need to work at all to get my paycheck. Once you realize the incentives and rewards don't make sense it's hard to stay motivated.
I've started using LLMs more for writing, but #2 is how I use them. For example, today, I needed to write a complaint to a company. So I wrote it from my head. But it had too many characters, the online form had a character limit, and I'm a wordy MFer.
So I gave it to ChatGPT and asked it to make it more concise. And it did. Think it like halved the number of characters, while still keeping the salient points. But afterwards, I still edited it, fixing mistakes, adding context that I thought was much needed, adding my own "voice" back, etc. And then submitted it.
I still wrote the base myself, had the thing edit it, and then I edited the edit. I was engaged the whole way through.
I use the same approach. It's an excellent editor and brainstorming buddy.
Oftentimes, I can come up with bullet points but hit a wall turning those into a cohesive paragraph of sentences. With some LLM assistance, I'm able to work through the writer's block and write out what I want to say!
It may never happen just due to the sheer size of LLMs, but this “your own voice” thing is why I would love a local-on-device version of an LLM in my phone that can learn from me and my own style, and keep improving over years.
On a similar note, I believe my Siri has gotten better and better over the years at understanding my accent and correctly interpreting what I’ve said, but only because I constantly corrected it whenever it misunderstood what I said to it. (Although it has kinda plateaued or gone backwards in recent years, I wonder if it still “learns” or if there’s too much AI crammed into it and it’s lost that ability these days)
The study of the pdf itself does indicate that they used multiple approaches, although it's also very long so I haven't been able to read through it thoroughly as of yet. Hopefully I will this weekend, though I'm also not a psychologist or neurologist so who knows how well I'll understand the particulars.
And yeah, as a rule of thumb I try to use LLMs only when I don't need it. Effectively automating what I could be doing easily but would take more time and effort. Even then I take care that the (programming) structure it returns is properly maintainable within the actual context of our software. Sure, that one variable name is good now but in 10 years? That sounds like a stupid question if your unfamiliar with how much that can matter. But it does.
I'm curious how companies that are betting big in 'AI' (I really prefer the term LLM or ML, but that's how it's often marketed in those circles because investors...) will be holding up in a few years. Especially with software. It's easy to overestimate and underestimate the capabilities of LLMs. I've regularly seen even people who are otherwise intelligent consider it to reliable to my (personal) liking.
Although now that I think about, the people with more emotional intelligence seem to be more skeptical of it... A story for another time, but something to ponder upon.
I agree I'd like to know more. I do workshops for my company that teaches your method, because this is the best way. This is an assistant to your knowledge and learning. I don't want anyone to think that they don't need to know how to do their job or they don't need to learn something new.
Do these results increase the pressure on entry-level talent? If the Brain-to-LLM groups outperform the LLM-to-Brain groups, it seems to suggest that more experienced or senior talent will become increasingly valuable over time.
How can we expect entry-level professionals to grow if they’re not given the space to stretch, learn, and even fail?
I mean, the same is true of being able to do mental arithmetic - because we now all have calculators in our pockets, the average person today is way less skilled at adding, subtracting, multiplying, etc. than the average person forty years ago. But that doesn't mean we are on average less cognitively capable, it means we are using our cognition on different tasks, and through that use building up different cognitive circuits.
We don't need to be forcing students to write essays without LLMs any more than we need to be forcing them to do long division on large numbers by hand. We need to be figuring out what the enabled brain work is with the LLM-capable part of the task taken care of, and start teaching students how to develop that skill.
I think calculators are an excellent analog to LLM's in this respect; Just because we have a device that outputs an answer to just about any arithmetic problem doesn't mean that learning arithmetic is somehow antiquated.
Learning to use a calculator is an essential skill for any person wanting to be even an armchair mathematician. But if you were to give a calculator to every elementary schooler and ONLY teach them how to do calculations on the calculator, you produce a generation of students who have no idea how to do math. If you want to be able to vet the answers a calculator gives, or to figure out which calculations will give you the answer that you're seeking, you need to learn the process of mathematics.
In education, we call this developing number sense. Without number sense, it's difficult to determine what operations need to be done, and if your answer is anywhere close to reality. Yes, when given an explicit math problem, you'll be able to solve it. But explicit math problems only exist as practice, and the real challenge is to learn how to apply different methods to problems to get a meaningful solution.
Same goes, I would say, for writing sense. Without it, you don't have the skill to assess what is written, or if the writing actually answers the question you were asking. Yes, essays in school seem tedious, boilerplate, and meaningless, but that's because they are practice. It's only after you've written dozens of essays that you're able to start to assess what phrases/structures work, and how to defend a thesis statement.
Just like a calculator, an LLM is best used by an already experienced writer who knows how the answer should be structured, and how to determine if an answer is meaningful and relevant to the problem.
If you're never taught to add by hand, then it's alarmingly difficult to figure out how addition actually works, and if you don't know how it works, then you won't be able to see when it's useful, and when it's not. I think having students write essays using chatGPT from the beginning would be as deleterious as letting students use a calculator from the beginning - a lack of fundamental skill and understanding of the craft.
I wouldn't call it that damning. All the paper really shows is that that using ChatGPT for short-form, time-constrained writing tasks appears to reduce cognitive engagement, lower short-term memory recall, and diminish participants' sense of ownership over their writing. These effects were most evident when users transitioned away from AI and continued to show diminished neural activity, suggesting potential dependency on external tools.
Which is like, no really? Anyone going through current era education industrial machines has been trained to focus on syntax and external revision giving higher scores—so banking on the tool to quickly get that done in 20 minutes makes sense. To hang a lot of this on EEG scans, which doesn't have any evidence afaik to tying to higher quality writing, seems more like someone wanted to run a cool EEG study and then picked something trendy to talk about (AI is making us more dumb!) than anything else.
Hell, I can barely remember 90% of the essays I have ever written and that was before LLM tools became widely available. I likely wouldn't have been able to quote from any of mine in these situations either.
I do agree that we shouldn't be introducing these tools without intentional thought and at an inappropriate time in a childs development—but the reality is that schools only see kids for 6-8 hours a day and they are going to be introduced to them anyways. I also agree that blindly copy-pasting from tools isn't going to prompt any memory retention, but neither would copy-pasting from any tool.
I write more about it here.
You bring up some very interesting points, I have some thoughts on these two especially:
Like always, proper use of a tool needs to be taught. However, my concern is that children will become demotivated if they begin with the LLM-assisted style of writing and then, at some later point, either due to difficulty, availability, or setting, have to make due without that and suddenly it requires effort and hard(er) work and complex, critical thinking.
The interesting part however begins if we try to deconstruct my three scenarios above:
Probably not an issue for much longer given the PhD-level (and improving) smartness of models today.
Smartphone + pocket power bank will be all you need for the efficient, local-first models of today and tomorrow. Seriously, this likely won’t be an issue unless you’re forced to write (?) something with pen & paper in a remote hut for a month.
Exam styles or job certifications might adapt, they might not, I don’t know.
So, am I overly concerned? Will LLM-improved essay writing be the new “you kids won’t always have a calculator in your pockets”—or will it shatter the cognitive abilities of today’s kids and future generations? I definitely can’t say, and probably no-one knows at this point.
The massive difference being that tools previously couldn’t let you copy-paste without further modification. Now, you can hand something in without having ever read it (nor the task description, in the extreme case) and it has a reasonable chance of being passable.
Also, the “a third of PDFs remain unread” stat from your post has both hilarious, concerning, and thought-worthy implications.
I agree that you at least had to do some further modification (paraphrasing at the very least), but I do think that it still has the same long term result (no real memory retention) even if you may remember it better in the short term.
That last stat is actually something I am hoping that LLM's and other tools will actually help with in the long-run, though I wonder if there is going to be a change in how people write paper's to make them more accessible to AI agents. I wonder if there will be some sort of SEO-style optimization to get your research discovered by them.
For what it is worth, while only being just a blog post and not a research counter-paper, I did find this critical examination of the study to at least have some points. As always, more research than a single study is needed before we really can start to tell something tangible on these kinds of things.
I struggled to parse the last sentence of the first paragraph, I think because there’s a stray word in there which wasn’t removed as the article was rewritten?
(Emphasis mine)
I ended up reading the rest of the article, but while it creates counter arguments, it doesn’t feel more substantive than a loosely-cited opinion and I was hoping for something more substantial or at least more objectively convincing.
Yeah this is one of my fears. I think that in general, we (humans) tend to be lazy. Even the most proactive people will be “lazy” in one thing or another - be it big or small - like using capsules to make a coffee instead of grinding the beans yourself, ordering uber instead of making dinner, using a phone instead of walking down the street to talk to your cousin, etc etc.
Basically, whenever we find a convenient solution - that is, a solution that lets us skip some work to reach the same end - we will tend to use it.
A solution that lets us skip thinking… Yeah I can see it. I can see people outsourcing a lot of things to the AI, from small things like helping them craft an email to big decisions of their life. Maybe not all the time, maybe not in a dystopian manner, but I can see some of it.
There’s still a lot of entropy between the user and the “output”. Like, you need to craft an email and you want the AI to do it for you, the time you spend giving context and explaining what you want to say, will make many people just not bother and write themselves. But if these tools get better and can get more context automatically - like how Google promises with Gemini in their products - if this entropy gets minimized, I can see this effect of “skipping” be more pronounced.
In the end, this just proves one thing I’ve been saying for years: we live in the age of convenience and information. With some taps, you can find out the name of that star or who did what in where in 1905. Our education systems, globally, should reform and pivot to teach and focus on critical thinking. There’s no need to memorize things anymore, we can find out about them with the thing in our pocket. Instead, we need to learn to think and know what to do with that information
I haven't seen many folks note that the participants only had 20 minutes to write their essays. The brain scan results don't seem surprising at all given that the LLM writers would need to copy/paste, perhaps mindlessly, to get the essay done, while the non-LLM writers had to think super hard. It's good to have emprical evidence for that, I suppose, but I look forward to more situated studies that look at how the cogntive impacts of how people use LLMs in more authentic contexts and for real-world purposes.
There's a lot of odd stuff about the methodology. The design of the study really gives the groups different tasks—the LLM based group more about creating and editing an essay and the other groups more given a writing assignment due to a tool removal if anything.
Honestly the entire idea that we can tell there is cognitive debt from assigning an SAT prompt in 20 minutes, to people who have been trained to focus on external revision in these tasks over internal, and then being surprised there were differences is a bit of a stretch. I find the paper constrains itself in its conclusions, but the rhetoric online surrounding it has misinterpreted it wildly.