Skip to content

The Reproducibility Crisis in Computational Science

Abstract

When a scientific result comes out of a program, the paper describing it is only part of the evidence; the rest is code and data that readers almost never see. The geophysicist Jon Claerbout at Stanford began making his group’s computations rebuildable in 1990, and in 1995 the statisticians Jonathan Buckheit and David Donoho reduced his idea to a slogan: a computational paper is “merely advertising of the scholarship.” The cases that followed showed what the advertising could hide: five protein structures retracted in 2006 because an in-house program had mirrored the data, cancer trials at Duke built on an analysis that outsiders could reproduce only by putting its errors back in, and chemistry scripts whose answers in 2019 depended on the operating system. Surveys found that most systems-research code could not be built from what was published, and a 2020 challenge asked scientists whether their own ten-year-old code still ran.

Advertising for the Scholarship

Claerbout worked on seismic imaging, where research aims at better pictures of the subsurface. His observation was that the result of such research is not an image of the subsurface but the software, with its parameter settings, that produced the image. From 1990 his group organised its work so that every figure in a book or report could be rebuilt from source. They published CD-ROMs through Stanford University Press holding the text together with a viewer: next to each figure, a reader could open the code that made it, “burn” the supplied illustration, and regenerate it on their own machine. Claerbout noted that the main beneficiary was usually the author: “it often seems that the greatest beneficiary of preparing the work in a reproducible form is the original author!”

Jonathan Buckheit and David Donoho, statisticians at Stanford, applied the discipline to wavelets. Their 1995 paper “WaveLab and Reproducible Research” described a free MATLAB library containing all the code needed to reproduce every figure in their published wavelet articles, and summarised Claerbout’s position:

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

The same paper predicted that the details of Claerbout’s method would not spread, because the internet had changed the means of distribution in the five years since he started. It did: code moved to websites and later to repositories, and the CD-ROM disappeared from the practice (see the Dead End below).

The term needs one distinction. Reproducibility in this sense means that the same code applied to the same data gives the published numbers. Replication means that an independent study of the question reaches the same conclusion. A result can be reproducible and wrong; Roger Peng and Stephanie Hicks put it as the question “Do I understand and trust this data analysis?”, which cannot be answered without the data and the code.

The Mirror-Image Proteins

Geoffrey Chang joined the faculty of the Scripps Research Institute in 1999, at 28, and won a Presidential Early Career Award the following year. His group used X-ray crystallography to solve the structures of membrane transport proteins, among them the bacterial transporters MsbA and EmrE. The structures were published in Science (2001 and twice in 2005), the Journal of Molecular Biology (2003) and PNAS (2004). Other labs had doubts; biochemical data suggested a different arrangement of the protein, but nobody could prove what was wrong.

In September 2006 Roger Dawson and Kaspar Locher published in Nature a higher-resolution structure of a related transporter that looked nothing like Chang’s. Chang went back to his data and found the cause in an in-house software utility: it had switched data so that the resulting models were mirror images of the correct ones. The Scientist reported that the fault lay in about ten lines of code. Chang retracted the three Science papers in December 2006 and the two others followed, five papers representing more than five years of work. “I hope people will understand that it was a mistake, and I’m very sorry for it,” he told Science. Corrected structures were published in late 2007.

The Duke Case

In 2006 a group at Duke University led by Anil Potti and Joseph Nevins reported in Nature Medicine that gene-expression signatures from cell lines could predict which chemotherapy drug a cancer patient would respond to. Duke started three clinical trials that assigned patients to treatment using the signatures, and the Moffitt Cancer Center in Tampa started a fourth.

Keith Baggerly and Kevin Coombes, biostatisticians at the MD Anderson Cancer Center in Houston, tried to reproduce the analysis and could not. Working from the public data, they found mislabelled samples, gene lists shifted by one row (“off-by-one” errors), and results that matched the published ones only when the same mistakes were made deliberately. Their letter appeared in Nature Medicine in November 2007; the full reconstruction, “Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology,” came out in the Annals of Applied Statistics in 2009.

The trials kept running. Moffitt ended its trial on 8 October 2009. A review panel set up by Duke found no reason to stop, and the trials were restarted in January 2010. The decisive fact in the end was not statistical: in July 2010 The Cancer Letter reported that Potti had falsely claimed to be a Rhodes Scholar. Duke put him on leave; he resigned in November 2010, two of the trials were terminated on 4 November 2010 and the third on 3 February 2011. Duke reviewed 40 of his papers and expected about two-thirds of them to be fully or partly retracted. In 2012 the Institute of Medicine published a 274-page report, Evolution of Translational Omics, on how such tests should be validated before they reach patients.

Peng and Hicks point to a limit of the remedy. The Cancer Letter’s reporting showed that the investigators had known about several of the errors and regarded them as “differences of opinion.” Mandatory code sharing would have made the errors visible sooner, but it would not have made their authors recognise them as errors.

Sorting Files

In 2014 Patrick Willoughby, Matthew Jansma and Thomas Hoye published a protocol in Nature Protocols for assigning the structure of a small molecule by comparing measured NMR chemical shifts with computed ones. It came with Python scripts that read the output files of the quantum-chemistry computation and turned them into predicted shifts. In 2019 a group at the University of Hawaii led by Philip Williams, checking the structures of compounds from a cyanobacterium, got different answers from the same scripts on different computers: 173.2 on Windows 10 and macOS Mavericks, 172.4 on Ubuntu 16 and 172.7 on macOS Mojave.

The cause was one function. The scripts read their input files with Python’s glob, which returns file names in whatever order the operating system lists them, and did not sort the list, so the files were processed in a different order on different systems. The Hawaii group reported the fault in Organic Letters in October 2019, in a paper whose title ended “Reveals a Glitch with the ‘Willoughby–Hoye’ Scripts for Calculating NMR Chemical Shifts.” Williams estimated that up to 150 to 160 published studies might have used the scripts. The fix was a call to sort().

Does It Build?

Computer science, where the experiment is the program, might have been expected to do better. Christian Collberg and Todd Proebsting of the University of Arizona tested that in 2013. They took 601 papers from eight ACM conferences (among them OSDI, SOSP, PLDI and SIGMOD) and five journals, asked for or downloaded the code behind those whose results depended on code, and allowed thirty minutes of programmer time to build each one. Only 32 percent built in that time. Counting builds that took longer, the figure rose to 48 percent, and counting cases where the authors stated that the code would build with reasonable effort, to 54 percent. The study, published in Communications of the ACM in 2016, measured only whether the code compiled, not whether it gave the published results.

Outside computing, the term “reproducibility crisis” came mainly from experimental fields. A Nature survey of 1,576 researchers in 2016 found that more than 70 percent had tried and failed to reproduce another scientist’s experiment, and more than half had failed to reproduce one of their own. Computational work was supposed to be the easy case, because a program can in principle be run again exactly.

Ten-Year-Old Code

Programs do not stay runnable. Nicolas Rougier of INRIA in Bordeaux and Konrad Hinsen of the CNRS in Orléans, who had founded the journal ReScience C in 2015 to publish independent reimplementations of published computational work, launched the Ten Years Reproducibility Challenge in 2019. Scientists were to take code from their own papers published ten or more years earlier and make it run again. The challenge was timed to the end of support for Python 2 on 1 January 2020. Thirty-five people entered, proposing 43 articles, and 28 reproducibility reports resulted; Nature reported on them in August 2020.

The reports read like a catalogue of how software decays. Rougier revived an image magnifier for the Apple II that he had written at 16 and published in a French hobbyist magazine, and needed new-old-stock floppy disks from 1993 to load it on a real machine. Hinsen failed with one of his own papers, because his early-1990s code was on magnetic tapes that nothing in reach could read any more. Sabino Maggi had to rebuild a virtual machine with MS-DOS 6.22, Windows 3.11 and an old Visual Basic to rerun a 1996 analysis, without knowing which Visual Basic version he had used, since the notebooks recording it had been destroyed by a basement water leak. Ludovic Courtès could compile his 2006 C code only after rolling half a dozen libraries back to older versions, a “downgrade cascade”; he and Hinsen recommended the Guix package manager, which rebuilds an entire software environment from source. Roberto Di Cosmo could not find the code for a 1998 paper on his own disks or his co-author’s, and found it in Software Heritage, the source-code archive he had founded in 2015.

Checking the Pandemic Model

The question reached politics in 2020. The Imperial College COVID-19 Response Team’s Report 9 of 16 March 2020 modelled how non-pharmaceutical interventions would change COVID-19 deaths and hospital demand. It was produced with CovidSim, a simulation code that built on work published in 2005 and 2006. Before its public release on GitHub the code was refactored, and the game programmer John Carmack, who examined it, reported that the bugs he found were “generally in paths that weren’t enabled or hit.” Stephen Eglen, a computational neuroscientist at Cambridge, then reran the released code for the CODECHECK initiative and reported on 1 June 2020 that the key results of Report 9 were reproducible, with small variations, mostly under five percent, that came from the now deterministic code, slightly different population input files and averaging over ten runs. The certificate covered the computation, not the epidemiology: Eglen checked that the code produced what the report said, not whether the model’s assumptions were right.

Dead End: The CD-ROM and the Executable Paper

Claerbout’s interactive CD-ROMs answered the problem with a physical medium, which limited them to whoever received a disc. Buckheit and Donoho already expected in 1995 that the specifics would not be widely adopted, and Peng and Hicks note that the internet replaced the scheme by letting anyone download code and data at almost no cost.

A second attempt put the computation inside the article. In 2011 Elsevier ran an Executable Paper Grand Challenge with the International Conference on Computational Science. First prize went to the Collage Authoring Environment from the Polish computing centre CYFRONET, which wove captured computations into an online article; second to SHARE, a portal of hosted virtual machines by Pieter Van Gorp and Steffen Mazanek; third to Matan Gavish and David Donoho for a universal identifier for computational results. Collage went into production in May 2013, when the journal Computers & Graphics published its first executable papers on ScienceDirect in a special issue on 3D object retrieval. What spread more widely were the less ambitious tools: code deposited in public repositories and archives, containers and package managers that pin a software environment, and computational notebooks (see Data Science and the Notebook), which have reproducibility problems of their own when cells are run out of order.

The spreadsheet side of the same problem, from the gene names that Excel turned into dates to the Reinhart and Rogoff growth figures, is covered in Spreadsheet Errors; the general fragility of the record in Link Rot and the Digital Dark Age.

📚 Sources