Skip to content

Famous Software Disasters

Abstract

Software fails differently from bridges. A bridge that collapses was overloaded or badly built; a program that kills people usually did exactly what it was written to do. This article collects the canonical cases in chronological order, from the Mariner 1 guidance error of 1962 to the Amazon outage of October 2025: the Therac-25 radiation overdoses, the AT&T long-distance collapse, the Patriot missile timing drift, the London Ambulance dispatch breakdown, the Pentium division bug, the Ariane 5 explosion, the Mars Climate Orbiter unit mix-up, the UK Post Office Horizon scandal, the 2003 Northeast blackout, Knight Capital’s $460 million in 45 minutes, the HealthCare.gov launch, the Amazon S3 typo, the Boeing 737 MAX, the Azure AD signing key, and the cloud outages of 2025 at Google and Amazon. Each one exposes a failure mode that still exists: reused code in a new context, removed safety interlocks, accumulated rounding error, identical software failing everywhere at once, mismatched assumptions between teams, untested deployment, and institutions that trusted computer output over human testimony.

Mariner 1

On 22 July 1962 an Atlas-Agena rocket lifted off from Cape Canaveral carrying Mariner 1, NASA’s first attempt to send a spacecraft past Venus. The booster flew normally until it began an unscheduled yaw-lift maneuver, and the guidance commands it was receiving made steering impossible and pointed it toward a crash. The range safety officer destroyed the vehicle 293 seconds after launch, six seconds before the spacecraft would have separated from its rocket.

The post-flight review board found two faults that only did damage together. The airborne beacon that supplied rate data to the ground guidance system dropped out four times, for periods between 1.5 and 61 seconds. And when the handwritten guidance equations had been translated into computer instructions, a superscript bar over one symbol was left out. The bar marked a smoothed value; without it the program worked with instantaneous values, and while the beacon was down those values produced wrong steering commands. Its twin, Mariner 2, launched on 27 August 1962 and passed Venus at a distance of 34,773 kilometers on 14 December.

Myth: Mariner 1 was destroyed by a missing hyphen

Press reports at the time called the missing character a hyphen, and Arthur C. Clarke’s The Promise of Space (1968) made it “the most expensive hyphen in history.” NASA’s summary of the review board’s finding names a different symbol, a superscript overbar in the handwritten equations, and a second, hardware cause: the failed rate beacon. The coding error mattered only while the beacon was out. See Myths and Misconceptions.

Therac-25

The Therac-25 was a radiation therapy machine built by Atomic Energy of Canada Limited (AECL). Its predecessor, the dual-mode Therac-20, had independent protective circuits and mechanical interlocks that physically prevented the electron beam from firing at high power without the target in place. The Therac-25 dropped most of those hardware safeguards; safety was enforced by software alone.

Between June 1985 and January 1987, the machine massively overdosed six patients in the US and Canada. At least three died from the overdoses. The principal defect was a race condition: if the operator entered and corrected treatment data quickly enough (something experienced operators did routinely) the machine could fire the full-power electron beam without the target in place, delivering doses estimated at 15,000 to 20,000 rads, where a typical single treatment was about 200. The machine reported only a cryptic “MALFUNCTION 54,” and operators, conditioned to frequent benign error messages, resumed treatment.

Nancy Leveson and Clark Turner’s investigation, published in IEEE Computer in 1993, became the founding case study of software safety engineering. Their conclusions reached beyond the bug: AECL had reused code from the older machines without systematic review, had no independent software testing, performed no failure analysis that included software, and repeatedly assured users the machine could not overdose. The deeper failures were institutional; see Secure by Design: A History of Software Security Engineering for the parallel history in security.

The AT&T Long-Distance Collapse

At about 2:25 p.m. Eastern time on Monday, 15 January 1990, Martin Luther King Day, a switching center in New York had a minor hardware problem. It took itself out of service for the four to six seconds its recovery routine needed, then told its neighbors it was back. AT&T’s long-distance network ran on 114 electronic 4ESS switches that exchanged such status messages over the Common Channel Signaling System 7 network. A neighbor that received the “back in service” message and, within milliseconds, a second message from New York corrupted its own data, detected the corruption, and reset itself. When it came back, it sent the same pair of messages to its own neighbors. The fault walked from switch to switch for about nine hours. Roughly half of AT&T’s long-distance callers heard a busy signal or the “all circuits are busy” recording; Bruce Sterling later put the number of uncompleted calls at about 70 million.

The defect had arrived in mid-December 1989, when AT&T loaded new software into the front-end processors of all 114 switches. Accounts that circulated among programmers within weeks, and the opening chapter of Sterling’s The Hacker Crackdown (1992), traced it to a C break statement inside an if clause nested within a switch: the break was meant to leave the if, but it left the whole switch, and the code went on to garble data it should have kept. Engineers stabilized the network that night by reducing the signaling load; the next day AT&T replaced the new software with an earlier version. Chairman Robert Allen apologized in an open letter: “We didn’t live up to our own standards of quality, and we didn’t live up to yours.”

The network had been designed so that no single switch could bring it down, and the switches monitored each other to isolate a faulty one. All of them ran the same recovery code, so the mechanism meant to contain a failure spread it. Police and telephone-company security investigators doubted the software explanation for months and suspected hackers; the crash is where Sterling begins his account of the 1990 crackdown on the computer underground (see The History of Hacking).

The Patriot Missile at Dhahran

On February 25, 1991, during the Gulf War, a Patriot missile battery in Dhahran, Saudi Arabia failed to track an incoming Iraqi Scud missile. The Scud struck a US Army barracks, killing 28 soldiers.

The US General Accounting Office investigation found the cause in arithmetic: the system counted time in tenths of seconds, and 0.1 has no exact representation in binary. The 24-bit fixed-point approximation introduced a tiny error each tick. The system had been designed for short deployments, but the Dhahran battery had been running for about 100 hours, long enough for the accumulated clock error of about 0.34 seconds to shift the range gate nearly 700 meters, so the system looked for the Scud in the wrong part of the sky and dismissed it. A corrected software version arrived in Dhahran the day after the strike.

The same class of fault, a counter that nobody expected to run that long, turned up in an airliner a quarter century later. In April 2015 the US Federal Aviation Administration warned that a counter in the Boeing 787’s power-control units overflowed after 248 days of continuous operation and could shut down all of the aircraft’s generators at once. Its interim remedy was the help-desk classic: switch the system off and on again at regular intervals.

The London Ambulance Service Dispatch System

At 07:00 on 26 October 1992 the London Ambulance Service (LAS), which served a resident population of about 6.8 million, switched its central control room to a new computer-aided despatch (CAD) system. The system was meant to take emergency calls, find the nearest free ambulance through automatic vehicle location, and mobilize it by data terminal, without paper. In the morning, with few calls, it coped. As the day got busier, crews pressed their status buttons late or out of sequence, vehicle positions went stale, and the system’s picture of the fleet drifted away from reality. It sent several ambulances to one incident or a distant one when a closer crew was free, and it generated exception messages faster than staff could clear them. Unanswered exceptions produced more exceptions, messages scrolled off the top of the screens, and callers waiting for an ambulance rang back, adding to the load. Average ring times for emergency calls peaked at around ten minutes. From the afternoon of 27 October the control room went back to the semi-manual procedure it had used before.

The inquiry report of February 1993 found that the computer “did not fail in a technical sense” on those two days: it did what it had been designed to do, but “much of the design had fatal flaws.” Neither the software nor its users had been ready. The software was incomplete and not fully tested, the hardware had never been tested under full load, staff had not all been trained, and the fallback to a second file server had not been tested at all. The procurement explained much of it. Consultants from Arthur Andersen had advised in 1990 that a packaged system would need about £1.5 million and 19 months; LAS management set a shorter timetable. The standing financial instructions required the lowest tender to be accepted unless there were “good and sufficient reasons to the contrary,” and the winning consortium of Apricot, Systems Options, and Datatrak bid £937,463, about £700,000 below the next bid. Systems Options quoted £35,000 for developing the dispatch software itself. Its experience lay in administrative systems for emergency services, not in command and control.

On 4 November the system failed in the technical sense too. Shortly after 2 a.m. it slowed and locked up, and the switch to the standby server did not work. Three weeks earlier a programmer had left in a piece of code that used up a small amount of the file server’s memory every time a vehicle was mobilized and never released it. The control room returned to paper and voice radio. Newspapers reported patient deaths blamed on the delays; the inquiry asked them to make clear that no coroner’s court had concluded that a late ambulance caused a death.

The Pentium Division Bug

Intel released the Pentium on 22 March 1993, 3.1 million transistors on 294 square millimetres, and showed it to the public at CeBIT in Hannover days later. In June 1994 Thomas Nicely, a mathematics professor in Virginia, noticed that his Pentium machine gave wrong reciprocals of large primes from about the ninth significant digit. He reported it to Intel on 24 October and to colleagues on 30 October; Intel admitted on 2 November that it knew of the fault. The cause was the lookup table of the chip’s SRT division algorithm: five of its 1,066 entries had been left at zero instead of 2. (Ken Shirriff’s 2024 analysis of the die found 16 empty cells, of which only those five can ever be read.) Intel first played down how many users would be affected, IBM contradicted it on 12 December, and on 20 December Intel agreed to replace every faulty chip. In January 1995 it booked a charge of $475 million for the recall. The bug was in a table of constants rather than in code, but the Munich computer scientist Thomas Huckle still ranks it second on his long list of software failures.

Ariane 5 Flight 501

On June 4, 1996, the first Ariane 5 rocket veered off course and self-destructed about 40 seconds after liftoff from Kourou, French Guiana, destroying the four Cluster science satellites it carried, a loss commonly put at several hundred million dollars.

The inquiry board’s report (the Lions report) traced the failure to a single software operation: a conversion of a 64-bit floating-point value (the horizontal bias, derived from the rocket’s horizontal velocity) into a 16-bit signed integer. The code had been written for the Ariane 4, whose flight profile kept the value within range. Ariane 5’s early trajectory built up horizontal velocity about five times faster; the value overflowed; the inertial reference system shut down and emitted a diagnostic bit pattern that the flight computer interpreted as flight data. The identical backup unit, running identical software, had failed identically 72 milliseconds earlier. The function that overflowed was an alignment routine that served no purpose after liftoff on the Ariane 5 at all; it kept running because it had on Ariane 4.

Info

Ariane 501 is the standard citation for two distinct lessons: reuse is not free (code carries the assumptions of its original environment), and redundancy does not protect against design faults (two identical computers running the same wrong program fail together).

Mars Climate Orbiter

On September 23, 1999, NASA’s Mars Climate Orbiter disappeared during orbital insertion at Mars. The mishap investigation found that ground software supplied by Lockheed Martin computed thruster impulse in pound-force seconds, while the navigation software at JPL expected newton-seconds, the metric unit specified in the interface documentation. The navigation software therefore underestimated the effect of every small thruster firing by a factor of 4.45, and the error in the trajectory built up over months of cruise. Instead of the planned closest approach of 226 kilometers, the spacecraft passed about 57 kilometers above Mars, too low to survive. The lost orbiter cost about $125 million; the wider Mars Surveyor ‘98 program (which also lost its companion Polar Lander months later) totaled roughly $328 million.

The failure is remembered as “the metric mix-up,” but the mishap board emphasized process: navigators had noticed anomalies for months, and the discrepancy was never escalated and resolved between the two organizations.

The Post Office Horizon Scandal

The slowest-moving disaster on this list killed no one in an instant and lost no rocket. From 1999, the UK Post Office rolled out Horizon, an accounting system built by ICL/Fujitsu, to thousands of sub-post offices. ICL Pathway had won the contract in May 1996, originally together with a benefits-payment card that the government cancelled in May 1999; by 2002 the system ran some 17,500 post offices with 40,000 counters and was billed as Europe’s largest civilian IT system, with more than 3.5 million lines of code. Horizon contained defects that could produce phantom accounting shortfalls. The Post Office treated the computer’s figures as unimpeachable evidence. Between 1999 and 2015 more than 900 subpostmasters were prosecuted for theft, fraud, and false accounting, about 700 of them by the Post Office itself in private prosecutions. People were imprisoned, bankrupted, and ostracized; some died before being cleared.

One of the victims, Alan Bates, who ran a sub-post office in Wales, found an unexplained £1,000 shortfall early in 2001, lost his contract in 2003, and began organising other victims; Computer Weekly published their cases in May 2009, and the Justice for Subpostmasters Alliance formed the same year. A 2019 High Court ruling, won by Bates and 554 fellow claimants, who settled for £58 million, found Horizon contained “bugs, errors and defects” capable of causing the shortfalls. On April 23, 2021 the Court of Appeal quashed the convictions of 39 subpostmasters, and in 2024 the UK Parliament passed the Post Office (Horizon System) Offences Act, quashing the convictions en masse. The affair is widely described as the most extensive miscarriage of justice in British legal history. The disaster was less the bugs than the legal presumption that the machine was right and the human was lying.

The 2003 Northeast Blackout

On the afternoon of 14 August 2003, the alarm system in the control room of FirstEnergy, an Ohio utility, stopped working. The last valid alarm arrived at 14:14 Eastern time. Nothing on the operators’ screens told them that the alarms had stopped; the system simply went quiet. At 14:41 the server hosting the alarm function failed and handed its work to a backup, which failed at 14:54. From 15:05 onward, three of FirstEnergy’s high-voltage lines sagged into overgrown trees and tripped one after another, and the operators, seeing no alarms, did not react. Shortly after 16:00 the overloads cascaded through the interconnected grid. The blackout covered an area with about 50 million people and 61,800 megawatts of load in Ohio, Michigan, Pennsylvania, New York, Vermont, Massachusetts, Connecticut, New Jersey, and Ontario. Some parts of the United States were without power for four days.

The alarm software was part of XA/21, an energy management system made by GE. Engineers needed about eight weeks to find the defect, and they reproduced it in GE’s laboratory in Florida only by slowing the system with deliberate delays while feeding it alarm inputs. It was a race condition: two processes contended for a shared data structure, a coding error let both write to it at once, and the corrupted data sent the alarm process into an infinite loop. Unprocessed events queued up until the server gave out. “We had in excess of three million online operational hours in which nothing had ever exercised that bug,” GE’s Mike Unum told SecurityFocus. GE sent a patch to more than 100 utilities running the system. The joint US-Canadian task force named the software failure as one cause among several, next to inadequate tree trimming and FirstEnergy’s poor awareness of its own grid; the alarm system had no function that warned operators when it was itself not working.

Knight Capital’s 45 Minutes

On August 1, 2012, Knight Capital Group, whose trading made up about a tenth of all volume in listed US equities, deployed new order-routing software to its eight production servers. The deployment reached only seven. The new code reused a configuration flag that, on the eighth server, activated Power Peg, an order-routing function Knight had stopped using in 2003 but never deleted. A 2005 code change had moved the counter that told Power Peg when a parent order was filled, and nobody had retested the old code since.

When the market opened, the eighth server fired millions of unintended orders across 154 stocks. In about 45 minutes, while trying to fill 212 customer orders, the router produced 4 million executions, and Knight lost more than $460 million by the SEC’s count (the firm’s own first estimate was $440 million), roughly four times its previous year’s profit. Before the open, an internal system had sent 97 automated e-mails referring to “Power Peg disabled”; they were not designed as alerts, and nobody acted on them. The firm survived only days as an independent company before an emergency rescue and eventual acquisition. The SEC’s 2013 order, which fined Knight $12 million under the market access rule, became required reading for deployment engineering: no second technician or written procedure to check the deployment, no controls that stopped the flood of orders, messages that went unread, and reuse of a flag whose old meaning was still wired into production. See High-Frequency Trading for the surrounding ecosystem.

HealthCare.gov

On 1 October 2013 the US government opened HealthCare.gov, the federal website through which residents of 36 states were to buy health insurance under the Affordable Care Act. It received 250,000 simultaneous users on the first day, about five times the number planned for, and outages began within two hours. The load was only part of the problem; the site also had core performance faults. On the first day, six people managed to submit an application and select a plan.

The Inspector General of the Department of Health and Human Services, reviewing the project in 2016, found the causes mostly outside the code. No one person at the Centers for Medicare & Medicaid Services (CMS) had clear authority over the project, business requirements kept changing until shortly before launch, and CMS told its main contractor, CGI Federal, to build on a NoSQL database platform that CGI’s managers said they had little experience with. CGI later admitted it “should have been more emphatic in warning CMS of the risks of launching.” Warnings about the schedule reached senior officials and the launch date stayed. After a “tech surge” of outside engineers in late October the site recovered; by the end of the first enrollment period on 31 March 2014, 5.4 million people had selected a plan through it. The estimated value of the six key contracts rose from $464 million to $824 million, and CGI’s own contract from $58 million to $207 million.

The Amazon S3 Typo

On 28 February 2017 the billing system of Amazon S3, the storage service of Amazon Web Services, was running slowly in the Northern Virginia region (US-EAST-1). At 9:37 a.m. Pacific time an engineer, following an established playbook, ran a command to take a few servers of a billing subsystem out of service. In Amazon’s words, “one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.” Among the removed machines were servers of the index subsystem, which keeps the location of every object stored in the region, and of the placement subsystem that allocates space for new ones. Both had to be restarted from scratch, and S3 in the region accepted no requests while they did.

Amazon had not fully restarted these subsystems in its larger regions for many years, and S3 had grown so much in the meantime that the restart and the integrity checks of its metadata took longer than expected. The index subsystem could serve reads again at 12:26 p.m. and S3 worked normally from 1:54 p.m., more than four hours after the typo. Everything built on S3 in the region failed with it: launches of new EC2 virtual machines, Lambda functions, and customer services such as Slack, Trello, and Quora. Amazon’s own Service Health Dashboard also depended on S3, so until 11:37 a.m. it could not mark the individual services as failing, and Amazon reported the outage through its Twitter account instead. The fixes were plain: the tool now removed capacity more slowly and refused to take a subsystem below its minimum, and the dashboard was moved to run across several regions.

The Boeing 737 MAX

On 29 October 2018, Lion Air flight 610 crashed into the Java Sea 13 minutes after taking off from Jakarta, killing all 189 people on board. On 10 March 2019, Ethiopian Airlines flight 302 crashed six minutes after takeoff from Addis Ababa, killing all 157. Both aircraft were new Boeing 737 MAX jets, and in both cases a faulty angle-of-attack sensor had fed a false high reading to the Maneuvering Characteristics Augmentation System (MCAS), software that pushed the nose down automatically. MCAS activated again and again while the pilots fought it, until the aircraft dived into the ground. The US Federal Aviation Administration grounded the type on 13 March 2019 and did not lift the order until 18 November 2020.

The 737 MAX carried larger engines than its predecessor, mounted further forward and higher on the wing, which changed how the aircraft handled near a stall; MCAS was added to compensate. The final report of the US House Committee on Transportation and Infrastructure (September 2020) traced how a small function became a lethal one. As first designed, MCAS could move the horizontal stabilizer by at most 0.6 degrees and only at high speed. In March 2016, after test flights, Boeing redesigned it to work at low speed as well, with up to 2.5 degrees of authority, and able to activate repeatedly. It read only one of the aircraft’s two angle-of-attack sensors, although a Boeing employee acting as an FAA Authorized Representative had asked in 2015 whether it was “vulnerable to single AOA sensor failures.” On 30 March 2016, hours after the redesign was approved, the FAA agreed to Boeing’s request to remove references to MCAS from the pilots’ operating manual. Boeing had a commercial reason to keep the change small: under a 2011 contract with Southwest Airlines, it owed a discount of $1 million per aircraft if the FAA required simulator training for pilots moving from the previous 737 to the MAX.

The Azure AD Signing Key

Cloud services have a single door: the login. At about 19:00 UTC on 15 March 2021, users around the world could no longer sign in to Microsoft Teams, Exchange Online, SharePoint, Dynamics, Xbox Live, or the Azure portal, and neither could users of third-party applications that relied on Microsoft’s identity service, Azure Active Directory, for authentication. The programs were running; nobody could get through the front door.

Microsoft’s preliminary root cause analysis described a housekeeping job. Azure AD signs its OpenID and other identity tokens with cryptographic keys, and “as part of standard security hygiene, an automated system, on a time-based schedule, removes keys that are no longer in use.” One key had been marked “retain” for longer than usual to support a cross-cloud migration. The automation had a bug that ignored the “retain” flag, and it deleted the key while it was still in use. Tokens signed with it no longer validated. Microsoft rolled the key metadata back at 21:05 UTC, but cached state kept some services failing; the impact window in Microsoft’s report runs until 09:25 UTC on 16 March. It was the second such failure in six months: on 28 September 2020 a faulty Azure AD update had slipped past Microsoft’s safe-deployment system and blocked sign-ins for about three hours.

The CrowdStrike Outage

On 19 July 2024 the security company CrowdStrike sent a routine content update to its Falcon sensor, a program that runs inside the Windows kernel to detect attacks. Machines that received it crashed and could not restart without manual repair; Microsoft counted about 8.5 million affected Windows devices, among them the systems of airlines, banks, hospitals, and broadcasters (see The Windows Story).

CrowdStrike’s root cause analysis of 6 August 2024 described a mismatch of one. In February 2024, sensor version 7.11 had introduced a new template type for detecting attacks through Windows interprocess communication. Its definition specified 21 input fields, but the sensor code that called the content interpreter supplied only 20 values. Tests and the first updates in the field all used a wildcard for the 21st field, so nothing ever read it. The update of 19 July was the first to specify a real value there. The content validator checked it against the 21-field definition and passed it; the interpreter, with no bounds check, read past the end of a 20-element array, and the kernel crashed. Because the update went to all customers at once rather than in stages, there was no small first group in which the fault could show up. CrowdStrike’s listed remedies were the textbook ones: check the field count at compile time, add the runtime bounds check, and roll out content in stages with canary testing.

Google Cloud’s Blank Fields

On 29 May 2025 Google added a quota check to Service Control, the component that decides for each API call to Google Cloud whether it is authorized and within its quota. The new code path had no error handling and, against Google’s own rules, was not behind a feature flag that could have switched it off. It stayed dormant until 12 June, when a policy change containing “unintended blank fields” was written to the Spanner database that holds quota settings. Because quota policy is global, the change was replicated to every region within seconds. Service Control read the blank fields, hit a null pointer, and crashed; restarted, it crashed again.

From 10:51 Pacific time, Google Cloud returned errors for API requests around the world. Google’s incident report lists more than seventy affected Cloud products plus Workspace services. Engineers had identified the cause within ten minutes, and the kill switch for the code path was ready after about twenty-five. In the large us-central1 region the restarting Service Control tasks all hit the overloaded database at once, since the software lacked randomized exponential backoff, and recovery there took about two hours and forty minutes. Some products did not report full recovery until the evening.

The AWS US-EAST-1 DNS Race

Amazon’s Northern Virginia region failed again on 20 October 2025, this time through a timing bug. DynamoDB, Amazon’s key-value database, publishes the addresses of its servers through DNS, maintained by automated “DNS Enactors” that apply plans produced by a planner. Two Enactors worked in an unlucky order: a delayed one applied an old plan over a newer one, and a clean-up process then deleted the old plan as stale. The result, Amazon reported, was “an incorrect empty DNS record” for the regional DynamoDB endpoint. From 11:48 p.m. Pacific time on 19 October, nothing could find DynamoDB in US-EAST-1.

DynamoDB itself was repaired by 2:40 a.m., but the region’s other services had been built on it. The system that manages the physical hosts for EC2 virtual machines had lost its leases and fell into a congestion collapse while recovering, so new instances could not start until the afternoon; network load balancers then removed healthy servers because health checks failed. Lambda, container services, Amazon Connect call centers, and sign-in to the AWS console were impaired. The event lasted from 11:48 p.m. on 19 October until 2:20 p.m. on 20 October, and outside Amazon it took down Fortnite, Snapchat, the messenger Signal, Amazon’s Ring doorbells, and in Britain Lloyds Bank and the tax authority HMRC.

What the Disasters Share

  • Reuse without revalidation: Therac-25 (Therac-20 code), Ariane 5 (Ariane 4 code), Knight Capital (a recycled flag). Code embeds the assumptions of its original environment, and the assumptions do not travel with it.
  • Safety margins removed because software “worked”: the Therac-25’s deleted hardware interlocks, MCAS’s single sensor, and the Post Office’s removal of human doubt.
  • Identical software failing everywhere at once: AT&T’s 114 switches, Ariane 5’s two inertial units, FirstEnergy’s primary and backup alarm servers, 8.5 million CrowdStrike machines, and every Google Cloud region within seconds of a global configuration change. Redundancy protects against random hardware faults, not against the same wrong program running on every copy.
  • Shared foundations: S3 in 2017 and DynamoDB in 2025 carried so much of Amazon’s own platform that their failure took dozens of other services along, including the dashboard meant to report it; Azure AD in 2021 showed that a working application is useless when the login in front of it fails. When thousands of companies rent the same region, one fault becomes everyone’s outage (see The Cloud Computing Era).
  • Accumulation: Patriot’s rounding error, the Boeing 787’s 248-day counter, London Ambulance’s memory leak, and Mars Climate Orbiter’s repeated small thruster errors were each individually negligible and collectively fatal.
  • Rare timing: the Therac-25, AT&T, and XA/21 defects all needed two events to arrive within milliseconds of each other, which is why testing had not found them and why their makers could point to years of trouble-free operation.
  • Schedule over readiness: London Ambulance and HealthCare.gov both went live on a fixed date with untested software, and both inquiries placed the main blame on management rather than programmers.
  • Warnings ignored: in almost every case operators, navigators, engineers, or auditors saw anomalies before the catastrophe, and the institution lacked a path for those observations to stop the system.

These cases are why memory safety, formal methods, secure-by-design engineering, and disciplined deployment practice are practical concerns, and why Y2K, the disaster that didn’t happen, is best read as the industry’s one great act of preventive maintenance.

📚 Sources