I thought this was an interesting blog post about how to interpret scientific studies. (But don't read the comments.) But in order to judge scientific literature for yourself, you need to read...
I thought this was an interesting blog post about how to interpret scientific studies. (But don't read the comments.)
But in order to judge scientific literature for yourself, you need to read lots of papers. It seems like a high bar? I think in practice, we often need to defer to other people who have read the literature. A lot depends on how much you trust them.
There is a formal way of writing up a review of the literature (meta-analysis) and there are also informal ways (writing a review or summary) and I'm not sure the formal way is better than the informal way.
Reading a lot of literature doesn't necessarily mean you know how to interpret the papers. Even when you can interpret the papers, it doesn't mean you know how to look out for bad design. With...
Reading a lot of literature doesn't necessarily mean you know how to interpret the papers. Even when you can interpret the papers, it doesn't mean you know how to look out for bad design. With that being said, I generally defer to experts in fields I don't have this depth of knowledge in, because there's a higher likelihood that they have read, interpreted, and can spot bad design.
There are additional benefits to meta-analysis over reviews and summaries. Perhaps the most important benefit is that you get to do statistics on a pool of statistics. This gives you characteristics and important information about the body of statistics. Additionally, doing statistics on a pool of statistics can increase your confidence that the effects were not due to chance (smaller p-value). In some cases, the effects are so small that you effectively have to pool results to meet statistical significance. In addition, meta-analyses can help to provide a framework for which to evaluate studies and results. Review papers often won't talk about meta narratives such as study design within a field, whereas this is often highlighted in meta-analyses. It can also sometimes help for people to logically understand the field to have summary tables explaining the included studies at a high level - once again these are commonplace in a meta-analysis but not necessary present in a review. Findings and methods of reporting on these studies are standardized and often distributed over a team of individuals rather than a single person - these methods help to reduce bias. One final point I'll make about meta-analyses which I find particularly important is the transparency into materials and methods. If the findings are suspect to me, or do not match up with what another researcher would be lead to believe, it's very easy to double check some of the search criteria.
Yes, another good reason to defer to experts. (But can you find good experts?) This pooling of statistical data seems fraught with error. In particular, are you including studies that are...
Yes, another good reason to defer to experts. (But can you find good experts?)
This pooling of statistical data seems fraught with error. In particular, are you including studies that are basically fraudulent? (Recently this was a problem with Ivermectin study meta-analysis.) Should the selection criteria include "I'm going to skip all the studies that look dodgy to me?"
I agree that it should be much easier to start from someone else's meta-analysis. Maybe think of it as a way to get familiar with the literature? Transparency helps you do that.
Luckily, the materials and methods will let you know exactly what criteria they used to include studies. I'd like to put emphasis on the final sentence in my last reply to you, copied here: If the...
In particular, are you including studies that are basically fraudulent?
Luckily, the materials and methods will let you know exactly what criteria they used to include studies.
Recently this was a problem with Ivermectin study meta-analysis
I'd like to put emphasis on the final sentence in my last reply to you, copied here:
If the findings are suspect to me, or do not match up with what another researcher would be lead to believe, it's very easy to double check some of the search criteria.
I agree that it should be much easier to start from someone else's meta-analysis.
What? I don't follow what claim you're making here and I never said anything about where its 'easy' to start.
Sorry, I was thinking that if you're dissatisfied by how a meta-analysis is done, one alternative would be to do your own, but you gave good reasons why modifying their work would be easier than that.
Sorry, I was thinking that if you're dissatisfied by how a meta-analysis is done, one alternative would be to do your own, but you gave good reasons why modifying their work would be easier than that.
I thought this was an interesting blog post about how to interpret scientific studies. (But don't read the comments.)
But in order to judge scientific literature for yourself, you need to read lots of papers. It seems like a high bar? I think in practice, we often need to defer to other people who have read the literature. A lot depends on how much you trust them.
There is a formal way of writing up a review of the literature (meta-analysis) and there are also informal ways (writing a review or summary) and I'm not sure the formal way is better than the informal way.
Reading a lot of literature doesn't necessarily mean you know how to interpret the papers. Even when you can interpret the papers, it doesn't mean you know how to look out for bad design. With that being said, I generally defer to experts in fields I don't have this depth of knowledge in, because there's a higher likelihood that they have read, interpreted, and can spot bad design.
There are additional benefits to meta-analysis over reviews and summaries. Perhaps the most important benefit is that you get to do statistics on a pool of statistics. This gives you characteristics and important information about the body of statistics. Additionally, doing statistics on a pool of statistics can increase your confidence that the effects were not due to chance (smaller p-value). In some cases, the effects are so small that you effectively have to pool results to meet statistical significance. In addition, meta-analyses can help to provide a framework for which to evaluate studies and results. Review papers often won't talk about meta narratives such as study design within a field, whereas this is often highlighted in meta-analyses. It can also sometimes help for people to logically understand the field to have summary tables explaining the included studies at a high level - once again these are commonplace in a meta-analysis but not necessary present in a review. Findings and methods of reporting on these studies are standardized and often distributed over a team of individuals rather than a single person - these methods help to reduce bias. One final point I'll make about meta-analyses which I find particularly important is the transparency into materials and methods. If the findings are suspect to me, or do not match up with what another researcher would be lead to believe, it's very easy to double check some of the search criteria.
Yes, another good reason to defer to experts. (But can you find good experts?)
This pooling of statistical data seems fraught with error. In particular, are you including studies that are basically fraudulent? (Recently this was a problem with Ivermectin study meta-analysis.) Should the selection criteria include "I'm going to skip all the studies that look dodgy to me?"
I agree that it should be much easier to start from someone else's meta-analysis. Maybe think of it as a way to get familiar with the literature? Transparency helps you do that.
Luckily, the materials and methods will let you know exactly what criteria they used to include studies.
I'd like to put emphasis on the final sentence in my last reply to you, copied here:
If the findings are suspect to me, or do not match up with what another researcher would be lead to believe, it's very easy to double check some of the search criteria.
What? I don't follow what claim you're making here and I never said anything about where its 'easy' to start.
Sorry, I was thinking that if you're dissatisfied by how a meta-analysis is done, one alternative would be to do your own, but you gave good reasons why modifying their work would be easier than that.