It's not that there's anything wrong with finding apparent correlations in a pile of data, the issue is with reporting them as if they were parts of the original hypothesis you designed the...
It's not that there's anything wrong with finding apparent correlations in a pile of data, the issue is with reporting them as if they were parts of the original hypothesis you designed the experiment to test.
It's fine to say, for example, "Well, blackberry tea doesn't prevent or treat kidney stones, and further research is needed on whether green tea is effective", but not "green tea prevents kidney stones with 95% confidence" when the stated hypothesis was "blackberry tea drinking prevents kidney stones".
The correlation can only be the basis for a new hypothesis test, not a proof of the original hypothesis.
I'd wager the prevalence of p hacking is one of the significant factors in the replication crisis. If you're data mining for "significant" results, there's a good likelihood you'll find something...
I'd wager the prevalence of p hacking is one of the significant factors in the replication crisis. If you're data mining for "significant" results, there's a good likelihood you'll find something if you look at enough factor combinations, but much of the time these are just statistical noise that cannot be replicated. These results can certainly be the starting point for future research but aren't likely meaningful on their own.
Original paper on plos|one
It's not that there's anything wrong with finding apparent correlations in a pile of data, the issue is with reporting them as if they were parts of the original hypothesis you designed the experiment to test.
It's fine to say, for example, "Well, blackberry tea doesn't prevent or treat kidney stones, and further research is needed on whether green tea is effective", but not "green tea prevents kidney stones with 95% confidence" when the stated hypothesis was "blackberry tea drinking prevents kidney stones".
The correlation can only be the basis for a new hypothesis test, not a proof of the original hypothesis.
I'd wager the prevalence of p hacking is one of the significant factors in the replication crisis. If you're data mining for "significant" results, there's a good likelihood you'll find something if you look at enough factor combinations, but much of the time these are just statistical noise that cannot be replicated. These results can certainly be the starting point for future research but aren't likely meaningful on their own.