Link Rot and the Digital Dark Age

Abstract

In 1986 the BBC condensed a nation onto two laserdiscs to celebrate the 900th birthday of the Domesday Book. Sixteen years later the discs were nearly unreadable while the 1086 parchment original remained perfectly legible at the National Archives. That inversion, new media dying faster than old ones, has a name: the digital dark age, coined by the librarian Terry Kuny in 1997. This article follows the evidence: link-rot studies that found half the web citations in US Supreme Court opinions dead, the deletion of GeoCities and the volunteer Archive Team that raced the shutdown, MySpace erasing twelve years of music, and NASA’s moon photographs rescued from tapes only four refrigerator-sized drives on Earth could still play.

A BBC Master AIV with Philips VP415 LaserVision player, the only system that could read the 1986 BBC Domesday discs. Image: simon.inns, CC BY 2.0, via Wikimedia Commons.

A Warning at a Library Conference

In September 1997, Terry Kuny, a consultant working for the National Library of Canada, delivered a paper at the IFLA library conference in Copenhagen titled “A Digital Dark Ages? Challenges in the Preservation of Electronic Information.” His argument ran against the optimism of the early web: paper survives neglect, digital data does not. Every file format, operating system, and storage medium has a commercial lifespan measured in years, and information that nobody actively migrates dies with its platform. “We are moving into an era where much of what we know today, much of what is coded and written electronically, will be lost forever,” he wrote, casting digital librarians as the monks of a new dark age, copying manuscripts while the infrastructure around them decayed.

The warning kept getting repeated by people hard to dismiss. In February 2015 Vint Cerf, co-designer of TCP/IP and by then Google’s chief internet evangelist, told the AAAS annual meeting that the 21st century risked becoming an information desert for future historians. His proposed remedy, which he called digital vellum, was to preserve not files but complete descriptions of the software and hardware needed to interpret them, an idea the emulation community had been practicing for years (see Emulation and Retrocomputing).

The Domesday Discs

The canonical cautionary tale was built with the best intentions and a large budget. Between 1984 and 1986 the BBC, with Acorn, Philips, and Logica, marked the 900th anniversary of William the Conqueror’s 1086 Domesday survey by compiling a new one: over one million people, most of them schoolchildren from more than 9,000 schools, wrote descriptions of their local area, which were combined with maps, statistics, photographs, and video into an interactive national portrait on two LV-ROM laserdiscs holding 300 MB per side alongside analogue video.

Reading the discs required a very specific machine: a BBC Master AIV with a SCSI interface and coprocessor, a trackball, and the Philips VP415 player, a system costing about £5,200. Few were sold outside schools and libraries. By 2002 working players had become rare, the LV-ROM format was long dead, and the project was a byword for digital obsolescence: the parchment original in Kew needed nothing but dry air, while the disc needed a company to keep making its player.

The rescue proved Kuny’s monks real. In 2002 the CAMiLEON project (University of Leeds and University of Michigan) read the data off the discs over the player’s SCSI port, captured the analogue video frames, and modified the open-source emulator BeebEm to recreate the whole system in software. In 2011 the BBC put the community disc’s 25,000 photographs and 150,000 pages of text back online as Domesday Reloaded; when that site closed in 2018, the content moved to the National Archives, where it now sits in the same institution as its 1086 ancestor. Volunteer efforts such as Domesday86 continue to preserve the original hardware and decode the discs from raw signal.

Measuring the Rot

For years “the web forgets” was an anecdote. Jonathan Zittrain turned it into a measurement. A 2013 Harvard study he ran with Kendra Albert and Lawrence Lessig checked every web link cited in US Supreme Court opinions and in three law journals: half the links in Supreme Court opinions since 1996 no longer led to the cited material, and over 70 percent of those in the Harvard Law Review were dead. Courts were building precedent on footnotes that no longer existed. The study’s practical output was Perma.cc, a service run by law libraries that archives a cited page permanently at the moment of citation.

A follow-up with John Bowers and Clare Stanton, published in 2021, examined roughly two million deep links in New York Times articles back to 1996: 25 percent were completely inaccessible, rising with age from 6 percent of links made in 2018 to 72 percent of links from 1998. The study also documented the quieter failure of content drift: links that still resolve but no longer show what the author cited. The paper of record, the authors noted, increasingly rests on evidence readers can no longer inspect. The systemic backstop is a single donation-funded nonprofit, the Internet Archive and its Wayback Machine, which courts, journalists, and Wikipedia editors now treat as the web’s memory of record.

The GeoCities Deletion

Jason Scott, founder of Archive Team, in 2017. Image: Dennis van Zuijlekom, CC BY-SA 2.0, via Wikimedia Commons.

GeoCities, founded in 1994, hosted the amateur web: tens of millions of hand-built homepages sorted into themed “neighborhoods,” complete with animated GIFs, guestbooks, and MIDI files. Yahoo! bought it in 1999 for about $3.6 billion in stock (see Dead End: Yahoo!). In April 2009 Yahoo! announced it was shutting the US service down, and on 26 October 2009 roughly 38 million pages of 1990s web culture went dark, an act the archivist Jason Scott compared to a museum burning its own collection to save on heating.

Scott had founded Archive Team earlier that year, a loose group of volunteers with the motto “We are going to rescue your shit.” Through the summer of 2009 their distributed crawlers pulled down as much of GeoCities as they could reach; a year later Archive Team released the salvage as a torrent of nearly a terabyte, and the Internet Archive preserved its own crawl. The artists Olia Lialina and Dragan Espenschied later mined the torrent for One Terabyte of Kilobyte Age, a long-running blog republishing screenshots of the rescued homepages. Archive Team has since repeated the drill dozens of times, for Google Video, Yahoo! Answers, and every other product sunset with user data inside (a pattern examined in Dead End: Google Reader). What the GeoCities rescue could not save is a measure of the method’s limits: everything unlinked, password-protected, or deleted before April 2009 is gone.

Losses Without a Rescue

When no volunteers are watching, the losses are quieter and larger. In March 2019 MySpace, once the largest social network on Earth, admitted that “as a result of a server migration project” every photo, video, and song uploaded before 2016 was gone. The estimate: about 50 million songs by 14 million artists, uploaded between 2003 and 2015, including early work by acts who existed nowhere else. Users had reported broken music links for over a year before the company confirmed the files were unrecoverable. No backup existed, or none that worked, and the archive of an entire era of independent music vanished as an IT footnote.

Institutional losses keep pace with corporate ones. The BBC’s Domesday near-miss had a television counterpart: broadcasters worldwide wiped master tapes for reuse into the 1970s, which is why dozens of early Doctor Who episodes survive only as audio. The end of Adobe Flash in 2020 orphaned two decades of browser games and net art overnight, a collapse and partial emulation rescue told in Dead End: Flash and the Plugin Web.

Reading the Moon Tapes

Lunar Orbiter 1’s August 1966 view of Earth over the Moon, recovered at full dynamic range from the original tapes by the Lunar Orbiter Image Recovery Project. Image: NASA, Lunar Orbiter Image Recovery Project, public domain, via Wikimedia Commons.

Sometimes the data survives and the machines do not. Between 1966 and 1967 NASA’s five Lunar Orbiter spacecraft photographed the Moon on 70mm film, developed it on board, and radioed the scans to Earth, where they were recorded on 2-inch Ampex FR-900 instrumentation tapes. The mission’s public images were degraded copies of copies; the tapes held the full quality, and they escaped destruction only because the JPL archivist Nancy Evans refused to let them be thrown out, storing some 1,500 tapes and four surplus FR-900 drives for two decades, part of the time in her garage.

In 2008 the engineer Dennis Wingo and the journalist Keith Cowing launched the Lunar Orbiter Image Recovery Project (LOIRP) to get the drives working again, setting up shop in a closed McDonald’s at NASA Ames that the team renamed McMoon’s. Restoring the 270-kilogram drives took scavenged parts and retired Ampex engineers, but in November 2008 the project released its first recovery: Lunar Orbiter 1’s 1966 photograph of Earth rising over the lunar horizon, at a dynamic range no one had seen in the intervening four decades. The recovered images went into NASA’s Planetary Data System, and because the originals resolved detail the 1960s prints had lost, they proved useful for comparing the Moon’s surface across 40 years. The margin of survival was one stubborn archivist and the last four compatible tape drives known to exist.

Dead End: Trusting the Medium

The recurring dead end in digital preservation is the belief that a durable storage medium is the same thing as preservation. The Domesday discs were physically fine in 2002; it was the player, the format, and the custom software that had died around them. The same logic dooms every “archival grade” product pitched since: gold CD-Rs whose dyes decayed anyway, magneto-optical cartridges whose drives disappeared, tape formats whose readers go out of production on a fixed cadence. A medium rated for a century is worthless after the last reader breaks, and readers are maintained by markets, which have no century-scale interests.

What actually works is the opposite bet, made by every effort in this article: assume all media and formats are temporary, keep multiple copies in multiple places, migrate them forward continuously, and preserve the ability to interpret old formats through emulation. Libraries formalized this as LOCKSS (“Lots Of Copies Keep Stuff Safe”), a Stanford-built system from 1999 in which libraries hold and cross-audit each other’s copies of electronic journals. It is the medieval solution: the Domesday Book of 1086 survives for the same reason ancient texts do, because institutions with long horizons kept copies and kept custody. The digital dark age is not a property of digital technology; it is what happens when nobody is assigned to prevent it.