All Data Deserves a Chance
By: Akshayan Vimalanathan
Not every experiment you conduct will be a success; this is a universal truth of scientific research. However, negative results–results that do not confirm an expected outcome or original hypothesis–should be embraced as well. These results urge the scientific community to explore alternative methods, teach us to critically examine our pre-existing beliefs, and lead to possible improvements in experimental design (1). Although frustrating, negative results play a crucial role in the self-correcting process of science.
The Stigma Surrounding Negative Results
Why are negative results looked upon unfavourably? One explanation stems from the misconception that negative findings reflect poorly on the competency of scientists. Many are worried the scientific community will attribute negative results to poorly designed studies and ill-informed researchers. Another explanation stems from the fact that negative results are associated with lower citation rates,1 making them unattractive to prominent journals. In the interest of cultivating their own reputations, journals are predominantly fixated on publishing exciting, statistically significant research with perceived high impact (2). Given the current scientific culture and its preference for positive findings, publishing negative results is an extremely difficult undertaking. Studies reporting negative findings are often not considered impactful enough for many journals and remain a low priority for publication (3). Overall, the dissemination of negative results is an uphill battle for scientists.
Why Publish Negative Findings?
Scientists trying to establish a successful research career face immense pressure to publish high impact papers. They are expected to generate a high publication output with an excellent citation rate to be competitive for promotions, tenure, and funding opportunities (1). Considering negative results yield less scientific interest and fewer citations than positive results, many scientists opt not to devote time and resources publishing negative results (1). This phenomenon–in which the outcome of a research study influences the decision to publish–is described as publication bias.
Publication bias refers to the fact that studies with statistically significant results are more likely to be submitted and published than work with non-significant results (4). With negative results not being attributed the same value as positive results, researchers have less incentive to submit negative results forward for publication. In addition, researchers feel the resources necessary to submit and publish negative findings far outweigh the perceived scientific contribution. Ultimately, these negative findings are buried and disregarded, leading to an overrepresentation of positive results that create bias in the scientific literature (5).
To address this bias, it is imperative for scientists to start publishing their negative findings. Failing to report negative results can skew the literature in favour of unfounded theories proposed by positive results. The inaccurate representation of data in the literature can have potentially grave consequences. Without negative results to inform the field, flawed theories remain uncorrected and continue to receive monetary support from agencies (6). Science cannot be self-correcting when information is missing from literature (7). Thus, a complete record of positive and negative results is needed to eliminate bias and paint a holistic picture. For instance, Dr. Andrew Wakefield, along with 12 co-authors, published an alarming finding that child vaccination increases the incidence of autism.8 The infamous paper triggered a decade-long decrease in child immunization and subsequently increased morbidity of preventable diseases, including measles, mumps, and rubella (9). Between 1999 and 2000, 13 studies with compelling negative results were published dismissing the notion that vaccines cause autism (9). Sadly, these negative results failed to attract the same attention as the original study. This unfortunate incident reminds us that the scientific community remains in the dark without the meaningful input of negative findings.
There is tremendous value in disseminating negative results to the broader scientific community. For instance, reporting negative results can prevent unnecessary replication of experiments that would have otherwise exhausted considerable time, effort, and resources (10). Reporting negative results can also encourage scientists to generate new hypotheses and inspire new directions of research using refined or novel methodologies (10). Future studies can undoubtedly be guided by the valuable insights of negative results.
A Welcomed Change
The current scientific culture tends to dismiss the importance of negative findings, as evidenced by publication bias. However, recent initiatives in the field of science suggest a cultural shift on the horizon. Several journals have emerged solely aimed at publishing negative results (3): The Journal of Negative Results in Biomedicine, the Journal of Negative Results – Ecology and Evolutionary Biology, and the Journal of Articles in Support of the Null Hypothesis. Elsevier, a well-established publisher, has also joined the growing movement in support of negative results by launching the open access journal New Negatives in Plant Sciences.10 Additional open-access journals that have contributed significantly to the dissemination of negative results include PLoS ONE and Disease Models & Mechanisms (1). The aforementioned journals evaluate research based only on its originality and competency, with no attempt being made to pre-judge significance (2). The journals are intended to encourage scientific debate by providing a platform for negative results to be heard and accessed by the entire scientific community.
Other initiatives include the announcements of two research prizes celebrating negative scientific results. The first prize, presented by the European College of Neuropsychopharmacology, is the ECNP Preclinical Network Data Prize, valued at €10,000 (US$11,800) and being offered for negative results in preclinical neuroscience (7). The second prize, presented by the Organization for Human Brain Mapping, is the OHBM Replication Award, valued at US$2,000 and being offered for best replication study (7). These two prizes are attempting to incentivize publication of negative results and replication studies respectively, signalling a shifting scientific culture.
It is encouraging to see initiatives from the scientific community address publication bias and promote reporting of negative results. An experiment should not have to show positive results to earn its place in the published literature (10). Science is clearly heading in a promising direction where both positive and negative results are being recognized for their merit. Negative findings provide valuable contributions to the literature that fundamentally move us towards unabridged science (1).
- Matosin N, Frank E, Engel M, et al. Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture. Disease Models & Mechanisms [Internet]. 2014 [cited 18 January 2018];7(2):171-173. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3917235/
- Curry S. On the importance of being negative. The Guardian [Internet]. 2018 [cited 18 January 2018];. Available from: https://www.theguardian.com/science/occams-corner/2015/mar/08/on-the-importance-of-being-negative
- O’Hara B. Negative results are published. Nature [Internet]. 2011 [cited 18 January 2018];471(7339):448-449. Available from: https://www.nature.com/articles/471448e
- Sutton A, Song F, Gilbody S, et al. Modelling publication bias in meta-analysis: a review. Statistical Methods in Medical Research [Internet]. 2000 [cited 18 January 2018];9(5):421-445. Available from: http://journals.sagepub.com/doi/pdf/10.1177/096228020000900503
- Duyx B, Urlings M, Swaen G, et al. Scientific citations favor positive results: a systematic review and meta-analysis. Journal of Clinical Epidemiology [Internet]. 2017 [cited 18 January 2018];88:92-101. Available from: https://www.sciencedirect.com/science/article/pii/S0895435616307508
- Weintraub P. The Importance of Publishing Negative Results. Journal of Insect Science [Internet]. 2016 [cited 18 January 2018];16(1):109. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5088693/
- Rewarding negative results keeps science on track. Nature [Internet]. 2017 [cited 18 January 2018];551(7681):414-414. Available from: https://www.nature.com/articles/d41586-017-07325-2
- Wakefield A, Murch S, Anthony A, et al. RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. The Lancet [Internet]. 1998 [cited 18 January 2018];351(9103):637-641. Available from: https://www.ncbi.nlm.nih.gov/pubmed/9500320
- Gerber J, Offit P. Vaccines and Autism: A Tale of Shifting Hypotheses. Clinical Infectious Diseases [Internet]. 2009 [cited 18 January 2018];48(4):456-461. Available from: https://www.ncbi.nlm.nih.gov/pubmed/19128068/
- Granqvist E. Why science needs to publish negative results. Elsevier [Internet]. 2015 [cited 18 January 2018];. Available from: https://www.elsevier.com/authors-update/story/innovation-in-publishing/why-science-needs-to-publish-negative-results