Volunteer-Maintained Infrastructure

Abstract

Much of the software the digital economy stands on is maintained by unpaid or barely paid volunteers: an encryption library funded by $2,000 a year in donations (Heartbleed, 2014), a cryptography tool whose author was going broke (GnuPG, 2015), eleven lines of JavaScript whose deletion broke build systems worldwide (left-pad, 2016), a logging library patched through the night by volunteers while half the internet was under attack (Log4Shell, 2021), and a compression tool whose exhausted maintainer was socially engineered into handing his project to an attacker (xz, 2024). This article follows the recurring crisis, the people at the bottom of the dependency stack, and the slow, partial arrival of money.

The Heartbleed logo, 2014, the first security bug with a name, a logo, and a website. Image: Leena Kurjenniska / Codenomicon, CC0, via Wikimedia Commons.

The Nebraska Problem

In August 2020 the webcomic xkcd published strip 2347, “Dependency”: a towering, teetering stack of blocks labeled “all modern digital infrastructure,” balanced on one small block near the bottom labeled “a project some random person in Nebraska has been thanklessly maintaining since 2003.” The cartoon became the standard illustration for a structural fact of modern software. Open source won: by the 2010s nearly every commercial product sat on a deep stack of free libraries. But “free” described the price, not the cost. Somebody still had to review patches, answer bug reports, and respond when a security hole appeared, and for a surprising number of load-bearing projects that somebody was one person, unpaid, in their spare time.

Economists have a name for what happens to shared resources nobody is obliged to maintain. Software adds a twist: code that works is invisible, and the better a library works, the less anyone thinks about who maintains it. The pattern was already visible in the 1990s (the Network Time Protocol ran for decades on the unpaid labor of one or two people), but it took a series of public failures, roughly one every two years from 2014 on, to turn an obscure sustainability worry into a policy topic discussed at the White House.

Heartbleed

On April 7, 2014, the OpenSSL project disclosed CVE-2014-0160, a bug in the TLS heartbeat extension that let anyone on the internet read up to 64 kilobytes of a server’s memory per request: session cookies, passwords, private keys. Google engineer Neel Mehta and the Finnish security firm Codenomicon had found it independently within days of each other. Codenomicon gave it a name, a bleeding-heart logo, and a website, heartbleed.com, inventing the branded vulnerability in the process. The bug had been introduced on December 31, 2011 in a patch by a German PhD student, and had shipped in every OpenSSL release for two years. At disclosure, about 17 percent of the web’s secure servers, roughly half a million machines, were exposed.

Then came the uncomfortable question: who maintained the encryption library securing most of the web? The answer, laid out by OpenSSL Software Foundation president Steve Marquess in a widely read essay, was a handful of part-timers and one person working full-time, funded by about $2,000 a year in donations. Marquess wrote that the mystery was not that a bug had slipped through, but that it had not happened more often. Within weeks the Linux Foundation launched the Core Infrastructure Initiative (announced April 24, 2014), with Google, Microsoft, Facebook, Intel, and other companies pledging money to fund exactly the kind of project OpenSSL was: critical, ubiquitous, and broke.

The Going Rate

Werner Koch, author and maintainer of GnuPG. Image: Chlor at English Wikipedia, CC BY-SA 3.0, via Wikimedia Commons.

Heartbleed put a spotlight on the people at the bottom of the stack, and what it found was poverty wages. In February 2015 ProPublica profiled Werner Koch, the German programmer who had written GnuPG in 1997 after hearing a talk by Richard Stallman, and had maintained it from his home in Erkrath ever since. GnuPG was the free implementation of the OpenPGP standard descended from Phil Zimmermann’s PGP; journalists used it, and Edward Snowden had used it to reach reporters securely. Koch had earned about $25,000 a year from the work since 2001 and was, in the article’s words, running out of money. The response was immediate: donations covered his $137,000 funding goal, the Core Infrastructure Initiative granted $60,000, and Facebook and Stripe each pledged $50,000 a year.

The rescue made a good headline, but the mechanism should give pause: a maintainer’s income depended on a journalist happening to write about him. Others were less lucky. Denis Pushkarev, sole maintainer of core-js, the polyfill library present on roughly half of the world’s top-1,000 websites, wrote in February 2023 that full-time maintenance had never paid him more than a few thousand dollars a month, had dropped to about $400 a month, and worked out to a few dollars per hour. His post, and the mixed sympathy and abuse it drew, changed little. Daniel Stenberg, who has maintained curl since 1998 as it spread into billions of devices, published one such demand in January 2022: a Fortune-500 company sent him a compliance questionnaire, with a deadline, requiring him to document his handling of the Log4j vulnerability. curl contains no Java and no Log4j, and the company had no contract with him. He offered to answer once they signed a support agreement, and heard nothing back.

left-pad

The dependency stack could fail without any attacker at all. On March 22, 2016, developer Azer Koçulu unpublished his 273 packages from the npm registry after a trademark dispute over one of them. Among the 273 was left-pad, eleven lines of code that padded a string on the left. It turned out to be a direct or transitive dependency of Babel, React tooling, and thousands of other packages, and its disappearance broke builds across the industry within hours (the incident is examined in The JavaScript Revolution). npm restored the package and tightened its unpublish policy. The lesson generalized far beyond JavaScript: the global software supply chain assumed that volunteers, bound by no contract, would keep their code available forever.

Log4Shell

On November 24, 2021, Chen Zhaojun of Alibaba’s cloud security team reported a flaw in Log4j, the Apache logging library embedded in a large fraction of the world’s Java software. Published on December 9-10 as CVE-2021-44228 and nicknamed Log4Shell, it scored the maximum 10.0 severity: a specially crafted string, written into any log, could make the library fetch and execute remote code via a JNDI lookup. Because logging is everywhere, the attack surface was everything from cloud services to the Mars helicopter’s ground software. Within days, security firms measured attack attempts in the millions; by mid-December nearly half of corporate networks had been probed. The US government ordered civilian agencies to patch by December 24, and the FTC announced in January 2022 that it intended to pursue companies that failed to update.

The fix fell on the Log4j maintainers, a small group of volunteers, several with day jobs, who worked through nights and weekends over the holidays while much of the internet was under active attack, and while angry users demanded to know why the bug existed at all. Maintainer Volkan Yazıcı answered on Twitter: “Log4j maintainers have been working sleeplessly on mitigation measures; fixes, docs, CVE, replies to inquiries, etc. Yet nothing is stopping people to bash us, for work we aren’t paid for, for a feature we all dislike yet needed to keep due to backward compatibility concerns.” The episode had one further casualty of note: China’s industry ministry suspended a partnership with Alibaba Cloud, penalizing the company because its engineer had reported the flaw to Apache rather than to the government first (a state-versus-vendor conflict examined in The Vulnerability Disclosure Debate). On January 13, 2022, the White House convened tech companies and open-source foundations to discuss securing open-source software, the first meeting of its kind.

The xz Backdoor

The darkest variant arrived in 2024, when the vulnerability was not a bug but a person. xz Utils, a compression library present in nearly every Linux system, had been maintained for years by one Finnish developer, Lasse Collin, for free. Starting in 2021 a contributor calling themself Jia Tan began submitting patches. Through 2022, accounts with names like “Jigar Kumar” that appeared nowhere else on the internet pressured Collin on the mailing list to hand over the project, complaining that patches sat unmerged. Collin answered candidly that he had not lost interest, but that his ability to care was limited by long-term mental health issues, and that Jia Tan might soon have a bigger role. By 2023 Jia Tan was co-maintainer; in February and March 2024 they shipped versions 5.6.0 and 5.6.1 containing a hidden backdoor (CVE-2024-3094, scored 10.0) that let a holder of a specific private key log into any affected machine’s SSH server as an attacker with root access. The multi-year patience, the sock-puppet pressure campaign, and the tradecraft in the payload all pointed to a professional operation, widely suspected to be a state intelligence service; the person or team behind “Jia Tan” has never been identified.

The backdoor reached the development branches of Debian, Fedora, and openSUSE, weeks from mass deployment. It was caught by luck. Andres Freund, a Microsoft engineer and PostgreSQL developer, was benchmarking a Debian test system when he noticed SSH logins consuming about half a second of unexplained CPU time and Valgrind throwing odd errors. He pulled the thread until he found the payload and posted his findings on March 29, 2024. The attack had needed no exploit at all. It had exploited the maintenance model itself: find the exhausted volunteer under the whole stack, and offer to help.

The Funding Response

Money did eventually start moving, in layers. The Core Infrastructure Initiative funded audits and key maintainers after 2014, then was succeeded in August 2020 by the broader Open Source Security Foundation (OpenSSF) under the Linux Foundation, whose Alpha-Omega project pays for security work in critical projects. Nadia Eghbal’s 2016 Ford Foundation report Roads and Bridges gave the problem its durable framing, open source code as unmaintained public infrastructure (its influence on the movement’s self-understanding is covered in The Open Source Movement). GitHub Sponsors (2019) and Open Collective built donation plumbing for individuals; Tidelift sells subscriptions and pays maintainers from the proceeds, an arrangement discussed in The Open Source Business Model.

The state arrived last. Germany’s Sovereign Tech Fund, created in 2022 under the federal economics ministry with a first-year budget of €13 million, pays maintainers directly for unglamorous infrastructure work: about €195,000 to curl, €200,000 to OpenSSH, over €1 million to the Python Package Index. The EU’s Cyber Resilience Act (2024) created a legal category of “open-source steward” while exempting non-commercial developers from product liability, after an outcry that early drafts would have made volunteers liable for their gifts. None of these mechanisms has changed the baseline: most of the load-bearing stack is still maintained the way xz was.

Dead End: Charity per Crisis

Each disaster produced a burst of generosity, and each burst faded. OpenSSL’s donations spiked after Heartbleed, then declined once the headlines moved on; the Core Infrastructure Initiative itself wound down within six years. Werner Koch was rescued because ProPublica wrote about him; Denis Pushkarev wrote his own appeal and stayed poor. The failure mode is structural. Attention-driven funding follows incidents, but maintenance is the opposite of an incident: it is the permanent, invisible work of making sure nothing happens. A model that pays out only when something breaks systematically underfunds the people whose success is measured by nothing breaking. The responses that have lasted are the ones that decoupled money from news cycles: employers assigning paid engineers to upstream projects, subscription pools, and state budgets. The tip jar, as a funding model for civilization-critical infrastructure, is the dead end.