I figured my classmates must be some kind of super-students capable of infinite focus. Then I peeked at some of the laptop screens around me and beheld—everybody else was doing exactly the same thing! We had all come to the library to study, and now all of us were goofing off. We were Goodharting ourselves. “Am I studying effectively?” is a hard question to answer. “How long did I spend in the library?” is way easier. Hours-in-library is a useful way to measure effectiveness-of-studying right up until it becomes a target—once you try to maximize your library hours, you find lots of ways to spend them doing anything but studying, and suddenly hours-in-library has nothing to do anymore with effectiveness-of-studying.
…
You know how lots of jobs force you to hang around and look busy until quitting time, even when there's nothing to do, because your dumb boss only cares about you being there from 9 til 5 and doesn't actually know what you're doing with your time? That's obviously a stupid policy, but what's even stupider is inflicting it upon yourself. Such is the terrifying power of Goodhart's Law: you can create your own system and game it too.
Regarding University Rankings: self-reporting creates all sorts of weird dynamics, as outlined in the article. But "third-party" evaluations are flawed too. I have an eastern European colleague in...
Regarding University Rankings: self-reporting creates all sorts of weird dynamics, as outlined in the article. But "third-party" evaluations are flawed too. I have an eastern European colleague in my lab who annually receives invitations to rank universities from his home country. He has never studied or taught there - basically, he has no clue about academics there, but his nationality fits.
Regarding library time: I haven't been a student in a while, but why do they have time to waste at the library in the first place? I used to review my classes of the day in three hours because that's all the time I had between extracurriculars and work.
I have a feeling I'm going to be rereading this article periodically for the next year or two, it's just such a clear elaboration on a relatable problem. I had never heard the term "Goodhart's...
I have a feeling I'm going to be rereading this article periodically for the next year or two, it's just such a clear elaboration on a relatable problem. I had never heard the term "Goodhart's Law" before, so the vocabulary alone has been a very valuable nugget of information.
I struggle with progressing long-term goals (ADHD), so I set concrete targets and build habits that aim for productivity, which eventually fall prey to Goodhart's Law. It can be a very fine needle to thread. Example: I set aside a dedicated block of time every week for writing. That's a butt-in-seat metric, which is quintissential Goodhart's Law material, so after the initial burst, progress slows in the way the article describes. Without dedicated butt-in-seat time, distractions consistently prevent progress, so the dedicated time is still an improvement. For that reason, I continued to lean into it.
But the article has made me see there are flaws to that too. It suggests I actually need to be finding a balance between the holistic and metric-driven approaches. First thing that comes to mind is regularly changing up targets - the existence of a target as a focal point is actually more important than its ability to measure anything.
Making time for things you want to do and "showing up" seems like the first step. I think writers sometimes set goals around the number of words written? (They might throw it out later, but it's...
Making time for things you want to do and "showing up" seems like the first step. I think writers sometimes set goals around the number of words written? (They might throw it out later, but it's practice.)
I run into Goodhart's law periodically and I am always left wondering the same thing: is there a way to avoid it? A strategy? Particularly (but not necessarily) in an organization? I was hoping to...
I run into Goodhart's law periodically and I am always left wondering the same thing: is there a way to avoid it? A strategy? Particularly (but not necessarily) in an organization?
I was hoping to see something like that in this blog post, but I was left hanging.
Personally, I feel that I've managed to avoid this trap most of the time because I generally see things holistically. So while I might put a small amount of energy into meeting some business target, I generally focus on the things that I think truly matter. (One exception, related to the author's library experience, is when I misguidedly took a job before finishing grad school, and then made arrangements with my employer to work on my thesis every Wednesday...and then very soon just goofed off every Wednesday). So I don't relate to hard to the "self-Goodharting" scenarios.
But I have mulled quite a bit how this plays out at work. Not everyone sees the whole picture all the time. Organizations need to align their non-omniscient workforces toward a goal. How do they do that? Are incentives or targets simply outright bad, because they will be gamed, curling the monkeys paw? Can they be designed holistically?
There are plenty of bad targets (with not without incentives attached). I've seen the potential pitfalls - salespeople selling unworkable solutions, products released before they are ready, accumulation of technical debt.
What is a good strategy to align the interests and work outputs of a group of people?
See also: productivity influencers who brag about how many books they've read and how early they wake up. Claiming they listen to an audiobook at 2x speed during their 5am commute.
I saw the same thing happening at the gym. There was always a guy there wandering around, lifting a weight or two, doing a squat, watching TV for a while, running for five minutes, checking his phone for a bit, and then wandering around some more. That guy thinks of himself as “spending two hours at the gym,” when he should actually think of himself as “spending 15 minutes working out.”
See also: productivity influencers who brag about how many books they've read and how early they wake up. Claiming they listen to an audiobook at 2x speed during their 5am commute.
From the blog post:
…
Regarding University Rankings: self-reporting creates all sorts of weird dynamics, as outlined in the article. But "third-party" evaluations are flawed too. I have an eastern European colleague in my lab who annually receives invitations to rank universities from his home country. He has never studied or taught there - basically, he has no clue about academics there, but his nationality fits.
Regarding library time: I haven't been a student in a while, but why do they have time to waste at the library in the first place? I used to review my classes of the day in three hours because that's all the time I had between extracurriculars and work.
I have a feeling I'm going to be rereading this article periodically for the next year or two, it's just such a clear elaboration on a relatable problem. I had never heard the term "Goodhart's Law" before, so the vocabulary alone has been a very valuable nugget of information.
I struggle with progressing long-term goals (ADHD), so I set concrete targets and build habits that aim for productivity, which eventually fall prey to Goodhart's Law. It can be a very fine needle to thread. Example: I set aside a dedicated block of time every week for writing. That's a butt-in-seat metric, which is quintissential Goodhart's Law material, so after the initial burst, progress slows in the way the article describes. Without dedicated butt-in-seat time, distractions consistently prevent progress, so the dedicated time is still an improvement. For that reason, I continued to lean into it.
But the article has made me see there are flaws to that too. It suggests I actually need to be finding a balance between the holistic and metric-driven approaches. First thing that comes to mind is regularly changing up targets - the existence of a target as a focal point is actually more important than its ability to measure anything.
Very exciting insight in this article!
Making time for things you want to do and "showing up" seems like the first step. I think writers sometimes set goals around the number of words written? (They might throw it out later, but it's practice.)
I run into Goodhart's law periodically and I am always left wondering the same thing: is there a way to avoid it? A strategy? Particularly (but not necessarily) in an organization?
I was hoping to see something like that in this blog post, but I was left hanging.
Personally, I feel that I've managed to avoid this trap most of the time because I generally see things holistically. So while I might put a small amount of energy into meeting some business target, I generally focus on the things that I think truly matter. (One exception, related to the author's library experience, is when I misguidedly took a job before finishing grad school, and then made arrangements with my employer to work on my thesis every Wednesday...and then very soon just goofed off every Wednesday). So I don't relate to hard to the "self-Goodharting" scenarios.
But I have mulled quite a bit how this plays out at work. Not everyone sees the whole picture all the time. Organizations need to align their non-omniscient workforces toward a goal. How do they do that? Are incentives or targets simply outright bad, because they will be gamed, curling the monkeys paw? Can they be designed holistically?
There are plenty of bad targets (with not without incentives attached). I've seen the potential pitfalls - salespeople selling unworkable solutions, products released before they are ready, accumulation of technical debt.
What is a good strategy to align the interests and work outputs of a group of people?
See also: productivity influencers who brag about how many books they've read and how early they wake up. Claiming they listen to an audiobook at 2x speed during their 5am commute.
Well, I skipped that bit because we don't know what the guy at the gym is thinking or what he tells others. It's something we imagine.
Thank you for posting a link to this blog. I read about a dozen articles and very much enjoyed them.
Yeah, he’s a really good writer.