Later Roberts, who headed one of the Cochrane groups, did a systematic review of colloids versus crystalloids only to discover again that many of the trials that were included in the review could not be trusted. He is now sceptical about all systematic reviews, particularly those that are mostly reviews of multiple small trials. He compared the original idea of systematic reviews as searching for diamonds, knowledge that was available if brought together in systematic reviews; now he thinks of systematic reviewing as searching through rubbish. He proposed that small, single centre trials should be discarded, not combined in systematic reviews.
Mol, like Roberts, has conducted systematic reviews only to realise that most of the trials included either were zombie trials that were fatally flawed or were untrustworthy. What, he asked, is the scale of the problem? Although retractions are increasing, only about 0.04% of biomedical studies have been retracted, suggesting the problem is small. But the anaesthetist John Carlisle analysed 526 trials submitted to Anaesthesia and found that 73 (14%) had false data, and 43 (8%) he categorised as zombie. When he was able to examine individual patient data in 153 studies, 67 (44%) had untrustworthy data and 40 (26%) were zombie trials. Many of the trials came from the same countries (Egypt, China, India, Iran, Japan, South Korea, and Turkey), and when John Ioannidis, a professor at Stanford University, examined individual patient data from trials submitted from those countries to Anaesthesia during a year he found that many were false: 100% (7/7) in Egypt; 75% (3/ 4) in Iran; 54% (7/13) in India; 46% (22/48) in China; 40% (2/5) in Turkey; 25% (5/20) in South Korea; and 18% (2/11) in Japan. Most of the trials were zombies. Ioannidis concluded that there are hundreds of thousands of zombie trials published from those countries alone.
Others have found similar results, and Mol’s best guess is that about 20% of trials are false. Very few of these papers are retracted.
In this issue, Anaesthesia, publishes an overview of the experience of the journal in trying to unearth submitted randomised controlled trials (RCT) that appear to be false and those where the data lack credibility so blatantly that they can be called ‘zombies’
From the blog post:
Is zombie trial a term I should've heard before? Completely unknown to me, but it's used without explanation here.
I hadn’t heard of it either. It seems to be defined (sort of) here:
Hundreds of thousands of zombie randomised trials circulate among us