Wow, I've barely gotten started but I just want to mention how brilliant this section is: Double-blind research creation fully separated from the hypothesis creation as a possible solution?
Wow, I've barely gotten started but I just want to mention how brilliant this section is:
When they measure a dozen things and only pick their outcome variable at the end, that's not the garden of forking paths but the greenhouse of fraud.
When they do a correlational analysis but give "policy implications" as if they were doing a causal one, they're not walking around the garden, they're doing the landscaping of forking paths.
When they take a continuous variable and arbitrarily bin it to do subgroup analysis or when they add an ad hoc quadratic term to their regression, they're...fertilizing the garden of forking paths? (Look, there's only so many horticultural metaphors, ok?)
The bottom line is this: if a random schmuck with zero domain expertise like me can predict what will replicate, then so can scientists who have spent half their lives studying this stuff. But they sure don't act like it.
Double-blind research creation fully separated from the hypothesis creation as a possible solution?
Interesting article. I have two questions: Is there a statistical study demonstrating the correlation between market predictions and replicability? A lot of the article depends on there being...
Is there an official reference/sanction for the author of the article? Most of the papers cited weren't related to the particular experiment the author was a part of, and it seemed the author could be going rogue.
Again, interesting read, but my family always ask me for sourcing, and I couldn't find any on my second perusal. Thanks
Rules for thee, but not for me! /s I thought this was an interesting read as well, but at the same time, there’s a lot of faith you have to put in the author in order to take anything substantive...
I like the article, it just seems important to apply the same criteria to predictive markets that the author does to the social studies themselves.
Rules for thee, but not for me! /s
I thought this was an interesting read as well, but at the same time, there’s a lot of faith you have to put in the author in order to take anything substantive away.
Is there an official reference/sanction for the author of the article? Most of the papers cited weren't related to the particular experiment the author was a part of, and it seemed the author could be going rogue.
I took this piece as totally rogue, but I can’t actually back that up. The about page of the blog just contains a bunch of quotes and the Twitter profile is equally nondescript. (Just a profile picture which is Velásquez’s portrait of Democritus.)
To quote the post:
There's a general lack of seriousness hanging over everything—if an undergrad cites a retracted paper in an essay, whatever; but if this is your life's work, surely you ought to treat the matter with some care and respect.
What standard of "seriousness" do we apply to (I assume) anonymous economist’s blog posts?
What does it mean to predict what will replicate? I’m just starting my study as of this year and this interests me, do they mean “how well the paper will be recited?”
What does it mean to predict what will replicate? I’m just starting my study as of this year and this interests me, do they mean “how well the paper will be recited?”
The replication crisis has to do with whether another group of scientists can independently replicate the results of a published scientific work (given the same resources and methods). It comes...
The replication crisis has to do with whether another group of scientists can independently replicate the results of a published scientific work (given the same resources and methods). It comes down to the notion that science worth citing ought to be reproducible and often isn’t (sometimes for bad reasons).
Seems like eternal september has come for scientific publishing. How much you want to bet if you carried out the same study on hard sciences like physics, chemistry, geology, biology they'd...
Seems like eternal september has come for scientific publishing. How much you want to bet if you carried out the same study on hard sciences like physics, chemistry, geology, biology they'd display a similar set of issues?
That was an excellent, if depressing read. If Tildes had exemplary tokens for submissions I'd use mine on this one.
There's some more discussion here with the author (I think?): https://old.reddit.com/r/slatestarcodex/comments/iqphdv/whats_wrong_with_social_science_and_how_to_fix_it/
Does anyone know about the background of the author? I would assume its someone in the humanities field but it's very hard to find information. Are they a phd student, a former academic, or...
Does anyone know about the background of the author? I would assume its someone in the humanities field but it's very hard to find information. Are they a phd student, a former academic, or something else?
I think the author posted here: https://old.reddit.com/r/slatestarcodex/comments/iqphdv/whats_wrong_with_social_science_and_how_to_fix_it/ Something to do with the Slate Star Codex blog?
Wow, I've barely gotten started but I just want to mention how brilliant this section is:
Double-blind research creation fully separated from the hypothesis creation as a possible solution?
Interesting article.
I have two questions:
A lot of the article depends on there being strong correlations, and the only evidence I saw for it was the 'lay people aren't terrible at it line'. Both papers cited about the strength of prediction markets used 41 or fewer test papers, which is incredibly small as far as sample size. And this paper gives a prediction performance of 59% for lay people, which is also poor. I like the article, it just seems important to apply the same criteria to predictive markets that the author does to the social studies themselves.
Again, interesting read, but my family always ask me for sourcing, and I couldn't find any on my second perusal. Thanks
Rules for thee, but not for me! /s
I thought this was an interesting read as well, but at the same time, there’s a lot of faith you have to put in the author in order to take anything substantive away.
I took this piece as totally rogue, but I can’t actually back that up. The about page of the blog just contains a bunch of quotes and the Twitter profile is equally nondescript. (Just a profile picture which is Velásquez’s portrait of Democritus.)
To quote the post:
What standard of "seriousness" do we apply to (I assume) anonymous economist’s blog posts?
As serious as the author wants his readers to take him, which, by the article's tone is pretty serious. :)
What does it mean to predict what will replicate? I’m just starting my study as of this year and this interests me, do they mean “how well the paper will be recited?”
The replication crisis has to do with whether another group of scientists can independently replicate the results of a published scientific work (given the same resources and methods). It comes down to the notion that science worth citing ought to be reproducible and often isn’t (sometimes for bad reasons).
Thank you, makes sense!
Seems like eternal september has come for scientific publishing. How much you want to bet if you carried out the same study on hard sciences like physics, chemistry, geology, biology they'd display a similar set of issues?
That was an excellent, if depressing read. If Tildes had exemplary tokens for submissions I'd use mine on this one.
There's some more discussion here with the author (I think?): https://old.reddit.com/r/slatestarcodex/comments/iqphdv/whats_wrong_with_social_science_and_how_to_fix_it/
What a great post. Thanks a lot for sharing!
Does anyone know about the background of the author? I would assume its someone in the humanities field but it's very hard to find information. Are they a phd student, a former academic, or something else?
They are anonymous (or I suppose pseudonymous) AFAICT. I assume they are an economist based on this post and the rest of the blog.
I think the author posted here: https://old.reddit.com/r/slatestarcodex/comments/iqphdv/whats_wrong_with_social_science_and_how_to_fix_it/
Something to do with the Slate Star Codex blog?