Laugh out loud moment (after talking about studies that show that monkeys can use money to pay for sex): The overarching theme of the smallness of the contributions reminds me so much of the...
Exemplary
Laugh out loud moment (after talking about studies that show that monkeys can use money to pay for sex):
Plenty of these findings are interesting and some are useful (especially if you are a rich, lonely monkey).
The overarching theme of the smallness of the contributions reminds me so much of the challenges of caring about my work in grad school. Two things especially it brought to mind:
The first is The Illustrated Guide to a PhD -- a visual representation of how the contribution of your dissertation to the sum of human knowledge is a tiny addition to a very narrow area.
The second is the oft-repeated statement about academia, "the arguments are so bitter because the stakes are so low." Basically, academics will fight to the bitter end to defend positions that ultimately only matters to about 10 people in the entire world. One reason for the acrimony is that people are usually competing for a very small pot of money, so it can easily become a zero sum game. But I believe the bigger reason is that for someone put so much time and energy into something that is ultimately has so little impact, you have to trick yourself on some level into thinking that the work is Really Very Important.
My problem with grad school was that once I realized how small the contributions are in the scheme of things, I was not able to maintain that fiction for myself. Once I realized that most of being faculty (especially at research institutions) was writing grants and politicking for funding while your grad students do most of the actual work, I wasn't interesting in pursuing it as a career, so I wasn't motivate to publish much. My advisor was fairly hands off, my motivation was low, and it led to me being in grad school for a very long time before finally eeking out my degree.
Another part of the problem was that my particular field (software reliability and system architecture) doesn't have a set of reliable metrics that can be used to benchmark and compare findings. It's also very hard to do realistic software development experiments because writing software is expensive and most people won't do it for free. So the experiments end up being pretty trivial compared to the scope and scale of commercial software development. In the end, my field shares a lot of the problems the article mentions about psychology research.
Contrast this to an adjacent area of research -- processor architecture. I shared office space with a processor architecture group. Measuring the impact of different architectural changes relied on widely used benchmark workloads with well-defined metrics. Someone could tweak a cache algorithm, show that it was 10% faster, write a paper, and move on. Do enough of this, and expect to go work for Intel or AMD or whoever designing processors that are going to go into thousands of computers.
Having said all that, there are a few times when that degree is useful. Sometimes, I can get people to listen to me because I have the letters PhD after my name. I really am an expert in a few things, and that really did start with the foundation in grad school, followed by years of experience "walking the walk" in an industry setting. However, knowing all the things I've written above, I often find the respect offered (and sometimes demanded by) PhDs purely because of having the degree to be pretty laughable. I feel like that's the quiet part we're not supposed to say out loud.
So, given my cynical attitude, I will offer this advice to people considering PhD work (for free! just imagine what it's worth!). Note that this is based on my experience in a computer science / engineering program at a top-tier research university. Take it with however much salt you like:
When you are applying / interviewing
If you're going to grad school, know what you expect to get out of it going in. Talk to people in the program about the realism of those expectations.
When you meet potential advisors, ask lots of questions about how your time at the university is funded -- how much does the professor have to provide, how much do you have to teach, where does the professor get most of their funding?
The best way to have a successful and interesting time in grad school (and to get done quickly) is to be part of a group that has long-term grant funding for a largish project that you can carve a contribution out of.
If your advisor has to piecemeal your funding, you'll often get shifted from project to project before you can build up a set of results that can form the basis of a dissertation. In that case, establish a topic early on that you can pursue without any support from a funder, and try to cram that in with the work tied into the funding.
Internships and industry collaborations can be a great source of data that can form the foundation of a dissertation. So network and form those connections if you can.
Talk to a potential advisors current and former students - find out if they are "live at the lab" types, "9-5 types", etc. This is mainly about knowing what to expect going in.
When you are in grad school
Figure out what the norms for the important conferences are about paper structure, how much of a contribution is "enough", what it takes to get things published.
Know that rejections are pretty arbitrary and don't take them personally. I sat in the PC meeting for a top tier conference, and the first day was vigorous discussion about the merits of the papers. By the last day, the committee members were so tired of talking, they were basically like "meh, does anybody give a shit" and if someone said yes, they would accept it, if someone voiced a concern, they would reject it without discussion.
Any time you are going to put something in front of faculty (e.g. qualifying exams, comp orals, your dissertation defense), find out who the committee members grad students are. Bribe them with food, beer, or money to get them to listen to a practice talk and give you feedback. I did this for quals, which were strictly timed, and found out that one of my committee members didn't believe anyone could present at a rate of faster than one slide per minute. "They will make you delete slides from your talk before you start until you have 20 slides left." I had a bunch of progressive disclosure animations made as multiple slides that I collapsed into animations. It's not only operational things. Faculty will often have pet theories, things they like to ask about, etc. and getting a heads up on those things can be a big help.
Any time you are feeling down, explain your work to someone who knows nothing about the field. You always know why it's bad, all the things you could have done but didn't have time to, etc. But when you explain it to someone, you'll remember all the things you did do and you'll feel better about it.
I don't have too much to add, I think it's telling how many votes your comment has (plus the exemplary tag) without responses, I suppose many (like myself) really felt what you had to say, thank...
I don't have too much to add, I think it's telling how many votes your comment has (plus the exemplary tag) without responses, I suppose many (like myself) really felt what you had to say, thank you for saying it.
To pick one thing though, a bit tangential from the link I know, but I think your last point is super important in terms of overcoming imposter syndrome. Between having an advisor you feel secure with, a cohort that y'all can ping ideas off each other and hear what other groups are doing, and that outside perspective, there are hardly any entry points for the imposter feeling to sink in. This can be adapted to areas outside of academic research as well.
This whole debacle matters a lot socially: careers ruined, reputations in tatters, lawsuits flying. But strangely, it doesn't seem to matter much scientifically. That is, our understanding of psychology remains unchanged. If you think of psychology as a forest, we haven't felled a tree or even broken a branch. We've lost a few apples.
That might sound like a dunk on Gino and Ariely, or like a claim about how experimental psychology is wonderfully robust. It is, unfortunately, neither. It is actually a terrifying fact that you can reveal whole swaths of a scientific field to be fraudulent and it doesn't make a difference. It's also a chance to see exactly what's gone wrong in psychology, and maybe how we can put it right.
…
[P]sychologists have been producing studies where humans deviate from the rules of rationality for 50 years. We've piled up hundreds of heuristics, biases, illusions, effects, and paradoxes, and if you scooped out Ariely's portion of the pile, it would still be a giant pile. A world without him is scientifically a very similar world to the one we have now.
…
This isn't specific to Gino and Ariely; I think you could It’s-a-Wonderful-Life most psychologists, even the famous ones, without any major changes to what we know. This was also true the last time we discovered a prolific fraudster. Diederik Stapel, a Dutch social psychologist, faked at least 58 papers. I mean really faked: the guy eventually admitted he would open up a blank spreadsheet and start typing numbers. Unlike Gino and Ariely, there's no ambiguity here—Stapel’s entire scientific career got wiped out.
So what was the scientific fallout of Stapel's demise? What theories had to be rewritten? What revisions did we have to make to our understanding of the human mind?
Basically none, as far as I can tell. The universities where Stapel worked released a long report cataloging all of his misdeeds, and the part called “Impact of the fraud” (section 3.7 if you're following along at home) details all sorts of reputational harm: students, schools, co-authors, journals, and even psychology itself all suffer from their association with Stapel. It says nothing about the scientific impact—the theories that have to be rolled back, the models that have to be retired, the subfields that are at square one again. And looking over Stapel's retracted work, it's because there are no theories, models, or subfields that changed much at all. The 10,000+ citations of his work now point nowhere, and it makes no difference.
As a young psychologist, this chills me to my bones. Apparently is possible to reach the stratosphere of scientific achievement, to publish over and over again in “high impact” journals, to rack up tens of thousands of citations, and for none of it to matter. Every marker of success, the things that are supposed to tell you that you're on the right track, that you're making a real contribution to science—they might mean nothing at all. So, uh, what exactly am I doing?
…
All of this is pretty distressing, but it feels a little better when you remember that science is a strong-link problem. That's why you can disappear entire careers and shoot holes through the literature without losing anything. Fields are mostly flab, so you're unlikely to hit any vital organs.
…
In other sciences, paradigms get overturned when they stop being able to explain the data coming in. If your theory can't account for why Neptune is over there right now, it's going to lose out to a theory that can.
Unfortunately, “humans are biased,” “situations matter,” and “pick a noun,” are unfalsifiable and inexhaustible. Nobody's ever going to prove that, actually, humans obey the laws of optimal decision making all the time. Nobody will show that situations don't matter at all. Nobody is going to demonstrate that leadership, creativity, or “social cryptomnesia” don’t exist. And we're never going to run out of biases, situations, or words. It's horrifying to think, but these proto-paradigms could be immortal.
But immortal does not mean invulnerable. Another way that paradigms die is people simply lose interest in them, so our best ally against these zombie paradigms is boredom. And we've got plenty. Psychologists already barely care about the findings in their own field; that's why, when we hear about another replication massacre, we don't even bother to ID the bodies. We're hungry for something that makes us feel. A few decades from now, when a wizened Bloom asks his question again, we hope for a world where people pile into the comments with major discoveries. Or, better yet, a world where Bloom doesn't even have to ask in the first place, because the answer is so obvious. (Imagine a computer scientist asking Twitter, “Hey guys, anybody hear about any big breakthroughs in computer science in the past few decades?”)
Laugh out loud moment (after talking about studies that show that monkeys can use money to pay for sex):
The overarching theme of the smallness of the contributions reminds me so much of the challenges of caring about my work in grad school. Two things especially it brought to mind:
The first is The Illustrated Guide to a PhD -- a visual representation of how the contribution of your dissertation to the sum of human knowledge is a tiny addition to a very narrow area.
The second is the oft-repeated statement about academia, "the arguments are so bitter because the stakes are so low." Basically, academics will fight to the bitter end to defend positions that ultimately only matters to about 10 people in the entire world. One reason for the acrimony is that people are usually competing for a very small pot of money, so it can easily become a zero sum game. But I believe the bigger reason is that for someone put so much time and energy into something that is ultimately has so little impact, you have to trick yourself on some level into thinking that the work is Really Very Important.
My problem with grad school was that once I realized how small the contributions are in the scheme of things, I was not able to maintain that fiction for myself. Once I realized that most of being faculty (especially at research institutions) was writing grants and politicking for funding while your grad students do most of the actual work, I wasn't interesting in pursuing it as a career, so I wasn't motivate to publish much. My advisor was fairly hands off, my motivation was low, and it led to me being in grad school for a very long time before finally eeking out my degree.
Another part of the problem was that my particular field (software reliability and system architecture) doesn't have a set of reliable metrics that can be used to benchmark and compare findings. It's also very hard to do realistic software development experiments because writing software is expensive and most people won't do it for free. So the experiments end up being pretty trivial compared to the scope and scale of commercial software development. In the end, my field shares a lot of the problems the article mentions about psychology research.
Contrast this to an adjacent area of research -- processor architecture. I shared office space with a processor architecture group. Measuring the impact of different architectural changes relied on widely used benchmark workloads with well-defined metrics. Someone could tweak a cache algorithm, show that it was 10% faster, write a paper, and move on. Do enough of this, and expect to go work for Intel or AMD or whoever designing processors that are going to go into thousands of computers.
Having said all that, there are a few times when that degree is useful. Sometimes, I can get people to listen to me because I have the letters PhD after my name. I really am an expert in a few things, and that really did start with the foundation in grad school, followed by years of experience "walking the walk" in an industry setting. However, knowing all the things I've written above, I often find the respect offered (and sometimes demanded by) PhDs purely because of having the degree to be pretty laughable. I feel like that's the quiet part we're not supposed to say out loud.
So, given my cynical attitude, I will offer this advice to people considering PhD work (for free! just imagine what it's worth!). Note that this is based on my experience in a computer science / engineering program at a top-tier research university. Take it with however much salt you like:
When you are applying / interviewing
When you are in grad school
I don't have too much to add, I think it's telling how many votes your comment has (plus the exemplary tag) without responses, I suppose many (like myself) really felt what you had to say, thank you for saying it.
To pick one thing though, a bit tangential from the link I know, but I think your last point is super important in terms of overcoming imposter syndrome. Between having an advisor you feel secure with, a cohort that y'all can ping ideas off each other and hear what other groups are doing, and that outside perspective, there are hardly any entry points for the imposter feeling to sink in. This can be adapted to areas outside of academic research as well.
From the blog post:
…
…
…
…