The review of scientific literature is a necessary and delicate operation

One of the most difficult tasks of the scientific community is to review all the scientific literature published on a given topic, to verify the state of knowledge regarding the efficacy of a medical treatment. It is a difficult task because, although there are refined scientific tools capable of aggregating the data produced by very different studies and procedures useful for rigorously deciding which studies can be included in a meta-analysis and which should be excluded, if the literature analyzed presents bias of a systemic nature, these can be difficult to highlight, even with the numerical techniques available; in this way, the image about the efficacy or safety of a given treatment can be distorted, and the scientific consensus can suffer as a result.

In order to identify what is the average weight of the bias contained in the literature regarding the efficacy of therapies or other procedures, or more generally regarding the significance of the results obtained in a given scientific sector, the ideal would be to have studies which, discussing certainly false results or, specifically, certainly ineffective therapeutic methods, instead presented on average a certain statistical significance in favor of a false hypothesis, or, in the case that interests us most here, they measured a certain average efficacy for a therapeutic approach of interest. In other words, a scientific “null field” would be needed, a set of articles reporting well-described results from a methodological point of view, certainly false results, to verify whether, despite everything, they are presented on average as significant and true; the measurement of false positives described in the “null field” would be useful in inferring the degree of bias positive which then afflicts all the other results in a comparable sector, where, however, it is not said that the results described are false, and indeed could very well be true.

If, for example, we set out to verify the accuracy of published results about the efficacy of a certain surgical procedure, we might think of verifying how effective a surgical method is reported to be for the same conditions, but completely and certainly ineffective; if there is a residual effectiveness reported for the second, we could attribute it to several bias by the authors of the articles describing it in the literatureand this bias it could constitute a “minimum efficacy” that a really functioning method must have in order to be taken into consideration, given that certainly ineffective methods reach, in words and in literature publications, this level. But what could be, concretely, a good “null field” to be used for example in the clinic?

A recently published study gives us a great example for this, evaluating the reported effectiveness for a procedure that has the greatest probability of actually being ineffectiveand clarifying at least in part the origin of the bias which leads to attributing an efficacy superior to placebo in randomized and blinded clinical trials. This is a work published by the group of Ioannidis, of Stanford, which has found an excellent example of a null field in medicine: homeopathy. The researchers reasoned in this regard as follows: the absence of an active principle in homeopathic preparations precludes their functioning, unless new laws of physics and chemistry are described which, in addition to explaining homeopathy, must have the same explanatory power of modern science in its entirety, from black holes to the molecular biology of any living being, because the principles that would be contradicted by the new theory capable of explaining the functioning of homeopathic preparations are those at the basis of modern science. For this reason, any presumed efficacy of a homeopathic preparation reported in the literature reflects to date bias positive of all kinds and therefore, as argued by Ioannidis and his colleagues, the publications that report analyzes of clinical efficacy in randomized and blinded studies are perfect for assessing the extent of these bias and of various types of error in a “null field” of science.

Compared with placebo, analyzes of 50 randomized, blinded trials reported 0.36-fold greater standardized efficacy; it is a clinical effect by no means negligible, which shows the extent of bias of publication. The proof that it is indeed an artifact comes from the detailed analysis conducted by the authors: calculation errors, erroneous statistical procedures, deviations from correct practices of all kinds are present in a very substantial part of the works examined and all push towards the overestimation of the efficacy of homeopathy, as indeed has already emerged from independent studies. Furthermore, the more the works showed efficacy of homeopathy, the more they were published in Open Access journals in which the authors pay to publish, and the more they were also cited by predatory journals.

By finding suitable and more specific examples of other null fields, for example by referring to specific clinical conditions and measuring the effects of presumed efficacy also for other theories whose probability of being true is very low from a Bayesian point of view, it will perhaps also be possible to clean up many other artifacts of literature, referring not to pseudoscientific procedures, but perhaps of overestimated effectiveness for any type of interest or error. Meanwhile, homeopathy, with its clinical studies, represents an excellent scientific “null field” for establishing a minimum average efficacy that should be recorded for treatments that actually function in equivalent conditions; in the end, even the publications that claim to take pseudoscience as founded, as we can see, if treated in the appropriate way can be useful to science.