Skip to content

David Mills and NTP

Abstract

David L. Mills (1938–2024) was one of the quiet architects of the early internet, the man who built the Fuzzball routers that stitched together the ARPANET and NSFNET, chaired the body that shaped how internet gateways behaved, and then solved a problem almost nobody realized was a problem: making thousands of computers on a slow, jittery, unreliable network agree on what time it is. His answer, the Network Time Protocol (NTP), first sketched in 1979 and standardized through the 1980s, still keeps the clocks of essentially every server, phone, and laptop on Earth within a few milliseconds of Coordinated Universal Time. Mills was legally blind for much of his life and fully blind by 2022, and he wrote and maintained this load-bearing piece of civilization for decades with a tiny circle of collaborators. The story of NTP is both a triumph of engineering and a cautionary parable about how the most critical infrastructure of the digital age can end up depending on one or two under-funded volunteers.

The blind engineer who built the internet’s plumbing

David Lennox Mills was born on June 3, 1938, in Oakland, California. He had congenital glaucoma; a surgeon saved some of the vision in his left eye when he was a child, and he attended a school for the visually impaired in San Mateo. His sight held for decades but began worsening around 2012, and by 2022 he was fully blind, yet he continued to work on the protocol he had created, relying on assistive technology and collaborators to read and write code he could no longer see on a screen.

Mills earned a PhD from the University of Michigan and spent his career at the intersection of hardware, networking, and precision measurement. From 1986 to 2008 he was a full professor at the University of Delaware, and it was from Newark, Delaware, that he ran the reference implementation of NTP for most of its life. He authored 28 RFCs (the standards documents that define the internet) including two full Internet Standards. He died on January 17, 2024, at the age of 85.

Before NTP made him quietly famous, Mills was already an insider of the early internet. He built the Fuzzball, a software router (running on the DEC LSI-11 minicomputer) that served as one of the workhorses of the early NSFNET when it launched in 1985, for a time, Fuzzballs were the backbone. He chaired the Gateway Algorithms and Data Structures (GADS) Task Force, which worked out how the routers (“gateways”) joining networks should actually behave, and he became the first chairman of the Internet Architecture Task Force. In an era when a handful of people were inventing the rules of internetworking alongside figures like Vint Cerf and Bob Kahn and Jon Postel, Mills was one of the small group turning the ARPANET into a working global system.

Why keeping time on a network is unexpectedly hard

It sounds trivial: every computer has a clock, so just ask it. But cheap quartz oscillators drift, a typical PC clock can gain or lose seconds per day, and no two agree. You cannot simply have one machine broadcast the “correct” time either, because the message takes time to arrive, and on a packet-switched network that delay is variable and unpredictable: a packet might cross the country in 30 milliseconds or 300, and the delay there is rarely the same as the delay back. Ask “what time is it?” and by the time the answer reaches you, it is already wrong by an unknown amount.

Mills’s genius was to treat time synchronization as a statistical problem rather than a lookup. NTP repeatedly exchanges timestamped packets with several servers, measures the round-trip delay of each exchange, and estimates the one-way offset, then it does this over and over, filtering out the noise. Crucially, it does not trust any single source. NTP uses the intersection algorithm, a refinement of Marzullo’s algorithm, to find the smallest interval of time consistent with the majority of its sources, discarding “falsetickers”, servers whose reported time cannot be reconciled with the rest. The result, remarkable for something running over the open internet, is agreement to within a few milliseconds of UTC, and better than a millisecond on a local network under good conditions.

This is a different problem from the one Leslie Lamport famously addressed. Lamport’s logical clocks establish the order of events in a distributed system without any reference to real-world time; they answer “did A happen before B?” NTP answers the harder, physical question: “what is the actual wall-clock time, everywhere, at once?” Both turn out to be essential, and they solve genuinely different halves of the coordination problem.

The stratum hierarchy, explained

NTP organizes the world’s clocks into a tree of trust called the stratum hierarchy, where the stratum number is roughly your distance, in hops, from a real source of truth:

  • Stratum 0, the reference clocks themselves: atomic clocks, GPS receivers, radio time signals. These are not on the network; they feed a computer directly.
  • Stratum 1, the primary time servers, directly attached to a stratum-0 device. These are the internet’s authoritative clocks.
  • Stratum 2, servers that synchronize to stratum 1 over the network, and in turn serve others.
  • Stratum 3, 4, …, each level down draws its time from the level above, out to a limit of stratum 15; stratum 16 is the label for a clock that is unsynchronized.

Your laptop is probably a stratum-3 or -4 client. The design is deliberately decentralized and self-healing: there is no single master clock for the internet, just a vast, redundant mesh in which time flows downhill from many independent sources, and any node can cross-check its neighbors. It is one of the most elegant pieces of infrastructure most people have never heard of.

From a 1979 demo to load-bearing infrastructure

Mills began working on network time synchronization at COMSAT Laboratories in the late 1970s. The technology was publicly demonstrated in 1979 at the National Computer Conference and first documented in RFC 778 (1981). NTPv0 was implemented and described in RFC 958 (1985). The protocol then matured through a long, patient sequence of standards, almost all shepherded by Mills himself:

  • NTPv1: RFC 1059 (1988)
  • NTPv2: RFC 1119 (1989)
  • NTPv3: RFC 1305 (1992)
  • NTPv4: RFC 5905 (2010), still the current version

Over those decades NTP went from a research curiosity to something the modern world silently assumes. Accurate time is a hidden prerequisite for almost everything: TLS certificates are rejected if your clock is wrong enough to think they are expired or not yet valid; Kerberos authentication refuses tickets whose timestamps are too far off; distributed databases rely on synchronized clocks to order transactions; log files from thousands of machines are only correlatable if their timestamps mean the same thing; and high-frequency financial trading is regulated on the assumption that everyone’s clocks agree. NTP also carries the warning bit for leap seconds, the occasional one-second adjustments that keep atomic time aligned with the wobble of the Earth’s rotation, a small but persistent source of software bugs, in the same family of calendar-and-clock hazards as the Y2K crisis.

⚠️ Dead End: when the internet’s clock runs on a volunteer’s savings

Here is the uncomfortable irony. NTP synchronizes billions of devices and underpins security, finance, and telecommunications worldwide, and for much of its history it was maintained by, essentially, one man and then a second man, with almost no money.

While Mills ran the reference implementation from his university office, the long-term stewardship fell to Harlan Stenn, who became the protocol’s chief maintainer and for years was effectively its sole full-time developer. In 2011 Stenn founded the Network Time Foundation to try to put NTP’s upkeep on a sustainable footing. The reality was stark: development was done by volunteers, on essentially no budget, kept alive “by the good graces” of a few companies and individuals. At one low point the foundation had collected only a few thousand dollars from a few dozen donors. It took a public alarm, the Linux Foundation’s Core Infrastructure Initiative, created in the wake of the Heartbleed disaster, eventually channeling on the order of $84,000 a year to Stenn, and a $60,000 donation from VMware in 2015, to keep the lights on.

This is not a one-off. It is a recurring structural failure of the digital economy, and NTP is its textbook case. The same shape recurred with the Heartbleed bug in OpenSSL (a ubiquitous encryption library maintained by a skeleton crew), and again in 2024 with the xz-utils backdoor, where a burned-out solo maintainer of a compression library used everywhere was socially engineered into handing control to an attacker. The webcomic xkcd 2347, “Dependencies,” captured it perfectly: a towering, precarious stack of “all modern digital infrastructure” balanced on a tiny block labeled “a project some random person in Nebraska has been thanklessly maintaining since 2003.”

Why does this keep happening? Because infrastructure that works is invisible. A protocol like NTP succeeds precisely by never being noticed; the better it works, the less anyone thinks about who maintains it. There is no product to sell, no user to bill, no quarterly demo, just the endless, unglamorous labor of keeping a decades-old codebase secure and correct while the whole world free-rides on it. Markets fund what is visible and new; they systematically under-fund the load-bearing commons. The parallel to Jon Postel (who for years personally was the internet’s address and naming authority) is exact: the early internet’s culture of trusting a few brilliant, generous individuals produced miracles, and also produced single points of failure that we are still struggling to institutionalize. The lesson of NTP is not that Mills and Stenn failed. It is that they succeeded so completely, and so cheaply, that the rest of us forgot the bill was still coming due. This is a core problem of the open-source model and, more broadly, of how open standards get maintained once the excitement of inventing them has passed.

Fun Fact

Mills gave NTP’s protocol messages a set of colorful internal names that read more like a bestiary than a spec. A time source whose clock is so wrong that it must be discarded is officially a “falseticker,” while the trustworthy ones are “truechimers.” And the software routers Mills built to run the early NSFNET backbone were called “Fuzzballs”, meaning that, for a stretch in the 1980s, a meaningful fraction of the internet’s traffic and much of its sense of time both flowed through machines Mills had personally named after nothing in particular.

📚 Sources