At the end of May, BMC Psychology attended Replication and reproducibility in psychology. BMC Psychology is one of the newer additions to the BMC series portfolio, and following a successful launch in February 2013, is building momentum and on its way to be becoming one of a handful of established open access psychology journals, and the only one to use open peer review.

Psychology and the reproducibility crisis

Concerns that scientific results are often not reproducible is by no means limited to the field of psychology. Worries about the lack of replication in psychology, without any hard evidence as to the scale of the problem, led Brian Nosek and colleagues to launch the Reproducibility project in 2011. Last year the Open Science Collaboration published the results of this large-scale, collaborative effort to estimate the reproducibility of psychological science. The project (which conducted replications of 100 studies published in three psychology journals) reported that whilst 97 out of the 100 studies originally reported statistically significant results, only 36 % of the replications did so, with a mean effect size of around half of that reported in the original studies. Other projects have since sought to tackle the same issue, with replication rates differing considerably across projects (read more here).

So should we despair at these findings? Or take this as an opportunity to help overcome the challenges of low reproducibility? Such was the topic of the #PsycDebate, which drew an audience of psychologists, students and researchers to the replication and reproducibility in psychology event.

The publication of that Science paper is revolutionary for our discipline

The debate was introduced by Daryl O’Connor (Leeds University) who suggested that “The publication of that Science paper is revolutionary for our discipline, it offers an opportunity to propel ourselves forward, to do things that other disciplines aren’t doing, and ultimately to improve scientific practice, research methods and transparency.”

applying scientific method to the process of science itself

Marcus Munafò (Bristol University) then began by discussing how the reproducibility crisis in psychology can be used as an opportunity to change. He highlighted how a culture in which researchers are encouraged to explore their data, rather than sticking to the original analysis plan, and may retrofit hypotheses to the data can be improved by “applying scientific method to the process of science itself”, thus enhancing quality control and improving rigor and reliability of science.

Roger Watt (University of Stirling) then discussed his view point that the ‘failure to replicate’ reported by Nosek and colleagues was not unexpected, largely due to the fact that p values are subject to sampling errors. He used a live statistical simulation to demonstrate that the ‘dance of the p values’ leads to a high rate of replication failures.

Dorothy Bishop (University of Oxford) then talked about the failure of psychologists to distinguish between hypothesis-testing and hypothesis-generating (exploratory) research, and how this may lead to the misuse of statistical tests. This problem is not new; it has been appreciated by statisticians for many years and was initially pointed out by Dutch psychologist Adriaan de Groot in 1956 (originally published in Dutch, translated recently). Dorothy discussed a number of potential solutions to the problem, including encouraging scientific rigor and publication of null results and replications, all of which are aspects of the BMC series ethos.

Chris Chambers (Cardiff University) then talked about another potential solution for distinguishing hypothesis-testing and hypothesis-generating research in psychology – pre-registration, whereby experimental methods and proposed analyses are pre-registered and peer-reviewed before data are collected.

The general consensus from the speakers and the attendees was the need to improve both research and publishing practices in psychology. We hope that BMC Psychology can be part of the solution! A common denominator across all areas of interest to the journal, as well as the BMC series as a whole, is rigor.

In a field which has been plagued by the under-reporting of both replications and null findings, BMC Psychology has a policy of publishing all sound science, thus aiming to reduce the impact of publication bias that may currently exist within the field of psychology. We do not make editorial decisions on the basis of the perceived interest of a study or its likely impact. Studies must be scientifically valid; for research articles this includes a scientifically sound research question, the use of suitable methods and analysis, and following community-agreed standards relevant to the psychology field.

Want to find our more? Check out BMC Psychology’s article collection Psychology: replication and beyond