The thorny problem of keeping time on the Internet

In 1977, David Mills, an eccentric engineer and computer scientist, took a job at COMSAT, a satellite company based in Washington, D.C. Mills was an inveterate tinkerer: He’d once built a hearing aid for a girl’s uncle and consulted for Ford about how to put paper tape computers into automobiles. Now at COMSATMills was involved in the ARPANET, the computer network that would become the forerunner of the Internet. A handful of researchers were already using the network to connect their distant computers and exchange information. But the fidelity of that exchanged data was threatened by a distinct deficiency: the machines did not share a single reliable synchronized time.

Over the decades, Mills had gained extensive experience in mathematics, engineering and computer science. In the early 1970s, as a lecturer at the University of Edinburgh, he had written programs that decoded shortwave radio and telegraph signals. Later, mostly for amusement, he had studied how grid clocks could drift several seconds on a hot summer day. (How much they moved depended not only on temperature, but also on whether the grid used coal or hydroelectric power.) Clock time, Mills learned, is the result of a relentless search for consensus. Even the times given by the more accurate “master clocks” maintained by the government of the world are composed of the readings of several atomic clocks. The master clocks, in turn, are averaged to help create international civil time, known as Coördinated Universal Time and initialized as UTC

To fix the time sync issue on the ARPANET, Mills created what programmers call a protocol, a collection of rules and procedures that creates a lingua franca for disparate devices. The ARPANET it was experimental and temperamental: electronics failed regularly, and technological bad behavior was common. The protocol of him sought to detect and correct those misdeeds, building a consensus about time through an ingenious system of suspicion. Mills prided himself on bizarre nomenclature, and so his clock synchronization system distinguished reliable “truechimers” from misleading “falsetickers.” An operating system called Fuzzball, which he designed, made the initial work easier. Mills called his creation the Network Time Protocol, and NTP soon became a key component of the nascent Internet. The programmers followed his instructions when they wrote the timing code for their computers. By 1988, Mills had perfected NTP to the point where it could synchronize connected computer clocks that indicated vastly different times to within tens of milliseconds, a fraction of the blink of an eye. “I always thought it was some kind of black magic,” Vint Cerf, an Internet infrastructure pioneer, told me.

Today we take global time synchronization for granted. It is fundamental to the Internet, and therefore to civilization. Vital systems – power grids, financial markets, telecommunications networks – rely on it to keep records and separate cause from effect. NTP works in conjunction with satellite systems, such as the Global Positioning System (GPS), and other technologies to synchronize the time across our many online devices. Time kept by precise, closely aligned atomic clocks, for example, can be transmitted via GPS to numerous receivers, including those in cell towers; these receivers can connect to NTP servers which then distribute the time to devices connected to each other from the Internet, almost all of which run NTP (atomic clocks can also directly supply the time to NTP servers). The protocol operates across billions of devices, coordinating time across every continent. The company has never been more in sync.

For decades, Mills has been the one to decide how NTP should work (although she disputes the suggestion that she acted with complete sovereignty). Bizarre, biting, authoritative, and sometimes opaque — “He doesn’t take fools gladly,” said a longtime collaborator — he was the father of Internet time. But his mandate is about to end. Mills was born with glaucoma. When he was a child, a surgeon was able to save some of the vision in his left eye and he has always worked using very large computer screens. About a decade ago, his vision started failing and now he is completely blind. Going through computer code and writing explanations and corrections has become incredibly boring. Drawing diagrams or composing complex mathematical equations is nearly impossible.

A couple of years ago, I visited Mills at his modest home in the Delaware suburbs. He and his wife, Beverly, have lived there since 1986, when Mills became a professor at the University of Delaware, a post he held for 22 years until his retirement. As we sat in his kitchen our conversation was regularly interrupted by an automatic voice announcing the time from the next room. The oven and microwave clocks were out of sync. Mills, who has a snow-white beard and wore a charcoal-colored fisherman’s sweater, keeps track of time using a talking wristwatch, which connects via radio signals to a master clock in Colorado.

He led me upstairs to his office, making our way slowly through the house looking for a series of memorized “navpoints.” At his desk, where a cat lay atop a crackling ham radio set, Mills sat down at the computer. He used the keyboard to call up a research paper he was working on, with suggestions for improving NTP (He asks his wife and daughter to proofread what he types.) As he used the arrow keys to scroll, the computer spoke to aloud. “This memo explores new security and protocol improvements,” said one voice. “Blank. Summary. Blank. One. Two. Colon. . . . Three. Three. Four. Four point one. . . .” It was soon lost.”I do what I can using the voice you hear,” Mills said. “But I look at myself and comment on the following: Man was made to do English composition with his eyeball.”

Technology doesn’t stop. The Internet continues to grow in both scale and complexity; even as its infrastructure ages, our world increasingly depends on its functioning. The continuous evolution of the Internet time synchronization system is essential. Yet Mills’ inability to contribute quickly to the NTP has weakened his authority over it. In his absence, only a few people seem to be able and willing to oversee critical but overlooked software. A contest has begun to influence the way clocks are kept in sync on the internet.

Mills was born in 1938 in Oakland, California, eleven years after the first quartz watch was developed and nine years before the first transistor was built. He took a steam train to a school for the blind in San Mateo and marveled at the engineers running it. In his teens, he became a model railroad enthusiast and radio amateur, communicating with friends and patching up Navy Seabees at the South Pole right up to their wives. His father, an engineer and salesman, co-founded National Oil Seal, a company that manufactured leak-prevention equipment. (“You may not know what it is, but there are at least two in your car engine,” his father told him, of the seals.) His mother trained as a pianist at the Toronto Conservatory of Music before staying in house to raise him and his two younger brothers.

The family moved, and Mills’ teachers did not always account for her visual impairment. Mills recalls an eighth-grade teacher telling him, “You’re never going to college,” a remark that was “like waving a flag in front of a bull,” he said. In 1971, Mills received a Ph.D. in computer science and communication sciences from the University of Michigan; after a two-year stint teaching in Edinburgh, he transferred with his wife and two children to the University of Maryland, which denied him tenure after five years. “It was the best thing that ever happened to me,” Mills said. He started working at COMSATwhere it had access to Defense Department funding, some of which went to the ARPANET. “It was a sandbox,” she later told an interviewer. “We have just been told: ‘Do good deeds.’ But good deeds were things like developing email and protocols. Part of the appeal of the time sync job, he told me, was that he was almost the only one doing it. He had the “little fiefdom” of him.

At NTP, Mills built a system that allowed for endless modifications and found joy in optimization. “The actual use of time information was not of central interest,” she recalled. The nascent Internet had few clocks to synchronize. But during the 1980s the network grew rapidly, and in the 1990s the widespread adoption of personal computers required the Internet to incorporate millions more devices than its early designers anticipated. Programmers created versions of NTP that ran on Unix and Windows machines. Others wrote “reference implementations” of NTP, open source code bases that exemplified how to run the protocol and that were freely available for users to adapt. Government agencies, including the National Institute of Standards and Technology (NIST) and the United States Naval Observatory, have begun distributing time measured by their master clocks using NTP

A free community of people around the world have set up their own servers to provide time through the protocol. In 2000, NTP servers answered eighteen billion time synchronization requests from several million computers, and in subsequent years, as bandwidth proliferated, requests to the busiest NTP servers increased tenfold. Time servers had once been “brightly lit in the United States and Europe, but dark elsewhere in South America, Africa and the Pacific,” Mills wrote in a 2003 article. “Today, the sun never sets or approaches never on the horizon on NTP.” Programmers began to take the protocol as a precondition: it seemed natural to them that synchronized time would be available reliably and easily. Mills’ small fiefdom was everywhere.

NTP works by telling computers to send tiny timestamped messages to time-control devices higher than them in a hierarchy. The top level of the hierarchy consists of servers tightly connected to extremely accurate clocks kept in close synchronization with Coördinated Universal Time. Time then flows, layer by layer, down to the machines at the bottom of the hierarchy, like ordinary laptops. The protocol keeps track of the moments that elapse while a time control message is sent, received, returned, and received again by the original sender. Meanwhile, a collection of algorithms, the “popcorn spike suppressor,” the “puff and puff filter,” sift through the data, spotting falseticker and truechimer, and instructing clocks how to adjust their times based on what they messages with date and time say.