Skip to content

The Space Shuttle's Two Software Teams That Were Forbidden to Communicate

Abstract

The Space Shuttle’s primary flight software (written in HAL/S by IBM) and its backup flight system (written independently by Rockwell, also in HAL/S but with its own compiler and design) were developed by teams that were deliberately kept separate. The backup team was given only the shuttle’s requirements, not the primary code, not its design, and not any contact with the primary software developers. The goal: if the two systems had the same bug, it would have to be a bug in the requirements, not a design error shared through communication. Across 135 missions the backup was never engaged in flight, so the scheme was never tested where it counted. Its most visible effect went the other way: the difficulty of synchronizing the two systems scrubbed the first shuttle launch in April 1981.

The Problem of Common-Mode Failure

In safety-critical systems, redundancy is the standard defense against failure: if one system fails, a backup takes over. But redundancy fails to protect against errors that appear in both the primary and backup systems: “common-mode failures.” If both systems are designed by the same team, following the same assumptions, reading the same documentation, and discussing the same implementation choices, they are likely to share the same bugs.

The Space Shuttle program’s flight software engineers understood this problem by the late 1970s. The primary flight software (approximately 420,000 lines of HAL/S code running on four of the orbiter’s five IBM AP-101 general purpose computers) controlled all aspects of ascent, orbital maneuvering, and re-entry. A software error in a critical flight phase could be catastrophic.

The solution was design diversity: a completely independent team, using different programming tools and prohibited from accessing the primary software, would implement the same requirements in a different system.

The Two Systems

Primary Avionics Software System (PASS): Written in HAL/S (High-order Assembly Language/Shuttle), a language specifically designed for the shuttle’s real-time requirements. Developed by IBM’s Federal Systems Division, whose on-board shuttle group passed to Loral in 1994 and to Lockheed Martin in 1996. The four computers running PASS voted on their results; a majority vote determined the commanded action. The system was 420,000+ lines across multiple releases over the program’s lifetime.

Backup Flight System (BFS): Also written in HAL/S, but independently by Rockwell (now part of Boeing), with a separate compiler (HAL/S-BFC, versus the HAL/S-FC compiler used for PASS) and an independent design. The BFS ran on the fifth AP-101 computer, loaded separately and activated only if PASS failed. The team was given the shuttle’s requirements documents and nothing else.

The separation was enforced operationally: BFS team members were not allowed to read PASS source code, attend PASS design reviews, or consult with PASS engineers about implementation choices. The only shared information was the specification of what the software had to do. NASA reinforced the split on its own side, assigning the Rockwell BFS contract to a different engineering directorate than the one managing the on-board software.

The Bug Heard Round the World

The first shuttle launch was scheduled for April 10, 1981. Roughly 20 minutes before liftoff, the backup computer failed to synchronize with two of the four primary machines, and the countdown was scrubbed.

The cause was a change made about a year earlier in an apparently unrelated part of the primary software. It opened a timing window, entered with a probability of about 1 in 67, in which powering up the primary avionics could leave critical cyclic processing one cycle out of phase. The window was hit when the first primary computer was powered up some 30 hours before launch; the mismatch only became visible when the backup came up shortly before liftoff. Columbia flew on April 12 after the systems were powered down and restarted.

The redundancy meant to protect against software failure had produced the program’s most famous software incident. Nancy Leveson, who studied the shuttle software in detail, judged the wider bet a poor one: the 1970s belief that independently developed versions would fail independently did not hold up, and the diverse backup probably bought less protection than its designers assumed. The BFS was never engaged to take over from PASS on any of the 135 flights.

The Quality of the Primary Software

The PASS software became famous in software engineering for its extremely low defect rate. Charles Fishman’s 1996 Fast Company article “They Write the Right Stuff” documented the process of the on-board shuttle group, by then 260 people working for Lockheed Martin in an office building across the street from the Johnson Space Center. Each of the last three versions of the 420,000-line program contained one error; the last eleven versions contained 17 errors between them. Fishman’s comparison was that commercial software of similar size would ship with thousands.

The team worked through formal inspections, extensive simulation testing, and a culture where finding bugs was rewarded rather than penalized. Review and specification work cost more than the coding did. PASS ran 135 missions over 30 years without a software failure that ended a flight.

The Apollo software tradition established before the shuttle program contributed to this culture: the computing heritage of NASA software was one of the few domains where software engineering’s most rigorous practices were commercially justified.

📚 Sources