Oh wow. The new results throw the original test results out the window. It's not about who you are innately, but what tools you come into the game with. Not surprising to me, but neat to see the...
Oh wow. The new results throw the original test results out the window. It's not about who you are innately, but what tools you come into the game with. Not surprising to me, but neat to see the results.
Yeah, it's really, really interesting. The main assumption at play was that things like social and economic differences weren't worth accounting for. But even the experimental design made...
Yeah, it's really, really interesting.
The main assumption at play was that things like social and economic differences weren't worth accounting for. But even the experimental design made assumptions, not just analysis of the data. The researchers assumed that all the kids could equally trust that they'd get a second marshmallow if they waited. But maybe they'd grown up in a household where money was tight and treats didn't always happen, even when promised.
After some time studying in college, I've noticed that with psychology a lot of things are now a mix of genetics and environment. It's interesting to see this get confirmed. It also makes me...
After some time studying in college, I've noticed that with psychology a lot of things are now a mix of genetics and environment. It's interesting to see this get confirmed. It also makes me wonder what other studies could be changed due to having people from different socioeconomic backgrounds. Just another reason researchers should be mindful of their samples.
You're absolutely right. There are so many potential cans of worms to be opened here. This actually excites me! I would love to be involved in retesting some of the old standards.
You're absolutely right. There are so many potential cans of worms to be opened here. This actually excites me! I would love to be involved in retesting some of the old standards.
One thing I've seen mentioned is the fact that a lot of psychology studies are done with college students (at least initially). That's a pretty specific slice of the population, and who knows how...
One thing I've seen mentioned is the fact that a lot of psychology studies are done with college students (at least initially). That's a pretty specific slice of the population, and who knows how accurate a slice it is for some of the questions they look at.
Yeah as a gen psych student part of our grade was doing 5 points of being a research subject, so we ended up taking part in about 2 or 3 studies. Also a lot of studies have some reward like a...
Yeah as a gen psych student part of our grade was doing 5 points of being a research subject, so we ended up taking part in about 2 or 3 studies. Also a lot of studies have some reward like a small gift card or something similar. I also wondered how this skewed data.
Interesting! I knew that students are often recruited, but had no idea it was sometimes part of a class grade. (I took general psychology in community college, before I transferred to a research...
Interesting! I knew that students are often recruited, but had no idea it was sometimes part of a class grade. (I took general psychology in community college, before I transferred to a research institution.)
Haha, fair point. ;) Although the article does point out that the kids in the original study were from a Stanford preschool, so that's still bias stemming from finding your study subjects "down...
Haha, fair point. ;)
Although the article does point out that the kids in the original study were from a Stanford preschool, so that's still bias stemming from finding your study subjects "down the hall" instead of making sure you have a representative sample.
More context: The study (behind a paywall) "Famed impulse control 'marshmallow test' fails in new research," The Guardian I thought this was a really interesting finding because it takes a very...
I thought this was a really interesting finding because it takes a very well known study and turns it on its head. It's a recent example of what's been called the "replication crisis", and I also think it's a good example of how assumptions and failing to account for variables can lead to faulty conclusions.
It seems we've got a lot of scientists here, or folks who've studied science in some way. Are there examples in anyone's field of this sort of thing? I know this the marshmallow test is psychology/sociology which might get dismissed by some as easily confounded, but I think scientists of any discipline are at risk of this sort of mistake. What steps do people take to avoid this sort of thing when designing and conducting experiments and analyzing the results? How do we deal with replicating results from studies that are costly or very time consuming (such as studies that follow children's development)?
While it's obviously not good that early studies were executed so poorly, I'm happy that replication crisis is even a thing. There is so much pressure in academia these days to publish new...
While it's obviously not good that early studies were executed so poorly, I'm happy that replication crisis is even a thing. There is so much pressure in academia these days to publish new original research; barely anyone gets resources to repeat prior studies. The problem is also that if replication is possible, it makes for really uninteresting reading. So until this whole publish or perish mess chills out a bit, we will likely continue relying on outdated or poorly executed 'science'.
I'm biased because I'm a sociologist, but I'd say that whenever you're working on an experiment that includes human participants, you have to have a good basic knowledge of sociology to be able to account for factors that might impact your experiment. A lot of it is just good practice, like how to select a good sample, but you'd be surprised how many scientists fail at this. I've read so many papers where the sample was crap, yet the authors tried to project the findings onto the entire population.
That's a good point about the pressure to publish something new. It kind of sounds like there needs to be a fundamental change in the system, so that replication studies are valued just as much as...
That's a good point about the pressure to publish something new. It kind of sounds like there needs to be a fundamental change in the system, so that replication studies are valued just as much as original ones, if not more. Perhaps having researchers start out their career with one... I'm not very familiar with getting PhDs or structure in academia. What do you think might help?
Honestly, what would help the most would be someone willing to fund these studies. Researchers pretty much do research on specific projects they get money for, so if no one is paying money for...
Honestly, what would help the most would be someone willing to fund these studies. Researchers pretty much do research on specific projects they get money for, so if no one is paying money for this, obviously the research won't be done because scientists need to eat, too. Right now, while replications obviously serve a wider interest of accuracy in science, they (especially if they are successful, which one hopes they mostly would be) are not super interesting for publication, meaning they don't add anything to researchers' CVs. And if you're an entity looking to commission a study, you probably will be interested in new and original research that is specific to your cause or problem you are facing, rather than a repeat of an earlier study.
Yeah, it makes sense money (and how impressive the work looks) would be the main stumbling blocks. This makes me wonder if there's a place for an organization solely built to provide grant money...
Yeah, it makes sense money (and how impressive the work looks) would be the main stumbling blocks. This makes me wonder if there's a place for an organization solely built to provide grant money for replication studies.
You mention entities commissioning studies-- perhaps replication studies could be promoted to them as a way of ensuring they're going down the right track for their cause or problem. "Save money by confirming results before basing more studies on this new finding."
As for CVs, I guess that would require a widespread change in how those are reviewed. If everyone expected a small "replication papers" section in addition to the impressive ones, maybe that would help? Plus, while we'd hope that most studies confirm results, sometimes they'll be wrong like with this marshmallow study. And then it seems very impressive (to me); not only did you do the work to repeat a study, showing commitment to accuracy in science, you analysed the past study, picked out issues with its design, created a better study, and implemented it.
EDIT: one problem I can already foresee is that folks who already have a tough time getting as many impressive papers might get stuck doing mostly replication studies, further putting them at a disadvantage when "publish or perish" is a common refrain.
Sociologist here: not surprising at all. It's something that I've wondered about myself, and this research confirms my suspicions. I'm really happy to see that awareness of people's social...
Sociologist here: not surprising at all. It's something that I've wondered about myself, and this research confirms my suspicions. I'm really happy to see that awareness of people's social circumstances is being factored in better in modern psychology research.
Oh wow. The new results throw the original test results out the window. It's not about who you are innately, but what tools you come into the game with. Not surprising to me, but neat to see the results.
Yeah, it's really, really interesting.
The main assumption at play was that things like social and economic differences weren't worth accounting for. But even the experimental design made assumptions, not just analysis of the data. The researchers assumed that all the kids could equally trust that they'd get a second marshmallow if they waited. But maybe they'd grown up in a household where money was tight and treats didn't always happen, even when promised.
After some time studying in college, I've noticed that with psychology a lot of things are now a mix of genetics and environment. It's interesting to see this get confirmed. It also makes me wonder what other studies could be changed due to having people from different socioeconomic backgrounds. Just another reason researchers should be mindful of their samples.
You're absolutely right. There are so many potential cans of worms to be opened here. This actually excites me! I would love to be involved in retesting some of the old standards.
Me too! So many old studies to go back and retest
One thing I've seen mentioned is the fact that a lot of psychology studies are done with college students (at least initially). That's a pretty specific slice of the population, and who knows how accurate a slice it is for some of the questions they look at.
Yeah as a gen psych student part of our grade was doing 5 points of being a research subject, so we ended up taking part in about 2 or 3 studies. Also a lot of studies have some reward like a small gift card or something similar. I also wondered how this skewed data.
Interesting! I knew that students are often recruited, but had no idea it was sometimes part of a class grade. (I took general psychology in community college, before I transferred to a research institution.)
Those are some smart kids if they're already in college when they do the marshmallow test.
Haha, fair point. ;)
Although the article does point out that the kids in the original study were from a Stanford preschool, so that's still bias stemming from finding your study subjects "down the hall" instead of making sure you have a representative sample.
More context:
I thought this was a really interesting finding because it takes a very well known study and turns it on its head. It's a recent example of what's been called the "replication crisis", and I also think it's a good example of how assumptions and failing to account for variables can lead to faulty conclusions.
It seems we've got a lot of scientists here, or folks who've studied science in some way. Are there examples in anyone's field of this sort of thing? I know this the marshmallow test is psychology/sociology which might get dismissed by some as easily confounded, but I think scientists of any discipline are at risk of this sort of mistake. What steps do people take to avoid this sort of thing when designing and conducting experiments and analyzing the results? How do we deal with replicating results from studies that are costly or very time consuming (such as studies that follow children's development)?
While it's obviously not good that early studies were executed so poorly, I'm happy that replication crisis is even a thing. There is so much pressure in academia these days to publish new original research; barely anyone gets resources to repeat prior studies. The problem is also that if replication is possible, it makes for really uninteresting reading. So until this whole publish or perish mess chills out a bit, we will likely continue relying on outdated or poorly executed 'science'.
I'm biased because I'm a sociologist, but I'd say that whenever you're working on an experiment that includes human participants, you have to have a good basic knowledge of sociology to be able to account for factors that might impact your experiment. A lot of it is just good practice, like how to select a good sample, but you'd be surprised how many scientists fail at this. I've read so many papers where the sample was crap, yet the authors tried to project the findings onto the entire population.
That's a good point about the pressure to publish something new. It kind of sounds like there needs to be a fundamental change in the system, so that replication studies are valued just as much as original ones, if not more. Perhaps having researchers start out their career with one... I'm not very familiar with getting PhDs or structure in academia. What do you think might help?
Honestly, what would help the most would be someone willing to fund these studies. Researchers pretty much do research on specific projects they get money for, so if no one is paying money for this, obviously the research won't be done because scientists need to eat, too. Right now, while replications obviously serve a wider interest of accuracy in science, they (especially if they are successful, which one hopes they mostly would be) are not super interesting for publication, meaning they don't add anything to researchers' CVs. And if you're an entity looking to commission a study, you probably will be interested in new and original research that is specific to your cause or problem you are facing, rather than a repeat of an earlier study.
Yeah, it makes sense money (and how impressive the work looks) would be the main stumbling blocks. This makes me wonder if there's a place for an organization solely built to provide grant money for replication studies.
You mention entities commissioning studies-- perhaps replication studies could be promoted to them as a way of ensuring they're going down the right track for their cause or problem. "Save money by confirming results before basing more studies on this new finding."
As for CVs, I guess that would require a widespread change in how those are reviewed. If everyone expected a small "replication papers" section in addition to the impressive ones, maybe that would help? Plus, while we'd hope that most studies confirm results, sometimes they'll be wrong like with this marshmallow study. And then it seems very impressive (to me); not only did you do the work to repeat a study, showing commitment to accuracy in science, you analysed the past study, picked out issues with its design, created a better study, and implemented it.
EDIT: one problem I can already foresee is that folks who already have a tough time getting as many impressive papers might get stuck doing mostly replication studies, further putting them at a disadvantage when "publish or perish" is a common refrain.
Full text: http://sci-hub.tw/http://journals.sagepub.com/doi/abs/10.1177/0956797618761661
It feels like almost every cool study I learned about in Intro Psych has been debunked in the past decade.
Sociologist here: not surprising at all. It's something that I've wondered about myself, and this research confirms my suspicions. I'm really happy to see that awareness of people's social circumstances is being factored in better in modern psychology research.