Skip to content

The Singularity and Superintelligence

Abstract

The idea that machines might one day improve themselves faster than people can follow has a short written history. The Bletchley Park codebreaker I. J. Good described an “intelligence explosion” in 1965 and called the first ultraintelligent machine “the last invention that man need ever make”; the mathematician and science-fiction writer Vernor Vinge named the moment the Singularity in 1983 and dated it, in a 1993 NASA symposium paper, to between 2005 and 2030; Ray Kurzweil moved it to 2045. From about 2000 a second line of thought, led by Eliezer Yudkowsky and the Oxford philosopher Nick Bostrom, turned the same forecast into a warning: a machine smarter than its makers would pursue its own goals, and nobody knew how to set them. After ChatGPT that warning reached the heads of the companies building the systems, who signed a one-sentence statement in May 2023 ranking the risk of extinction from AI with pandemics and nuclear war. Every dated prediction in this history so far has passed or been moved.

Vernor Vinge Orban
Vernor Vinge, who named the Singularity. Image: David Orban, CC BY 2.0, via Wikimedia Commons.

An Essential Singularity

The first use of the word in this sense is second-hand. In his memorial tribute to John von Neumann, published in 1958, Stanisław Ulam recalled a conversation that “centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.”

Good’s Explosion

Irving John Good, born Isadore Jacob Gudak in London in 1916, joined Alan Turing’s Hut 8 at Bletchley Park in May 1941 (see Alan Turing and the Enigma). After the war he worked as a statistician, and in 1965 he wrote “Speculations Concerning the First Ultraintelligent Machine”, published in Advances in Computers in 1966. Its argument took one paragraph. A machine that surpassed any person at every intellectual activity could also design machines, better than people could; “there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.” He thought it “more probable than not” that such a machine would be built within the twentieth century.

Good moved to Virginia Tech in 1967 and advised Stanley Kubrick on the computer for 2001: A Space Odyssey. According to his assistant Leslie Pendleton, an unpublished autobiographical note he wrote in 1998 had revised the forecast: he now suspected that an ultraintelligent machine would lead to the extinction of mankind.

Vinge’s Word

Vernor Vinge, a mathematician at San Diego State University and a science-fiction writer, took the step from Good’s premise to a limit on prediction. In a column in Omni in January 1983 he called the arrival of superhuman intelligence a singularity, a transition past which the future could no more be seen than the inside of a black hole. His 1981 novella True Names had already imagined people meeting in a shared computer world. At the VISION-21 symposium held by NASA’s Lewis Research Center and the Ohio Aerospace Institute on 30–31 March 1993 he gave the paper that carried the idea beyond science fiction, “The Coming Technological Singularity: How to Survive in the Post-Human Era”. Its abstract opened: “Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.” Vinge was specific: “I’ll be surprised if this event occurs before 2005 or after 2030.” He listed several routes, computers that “wake up”, networks, human-computer interfaces and biology, and argued that no ban could stop it, since the advantage of each step in automation was too great. He died on 20 March 2024.

Kurzweil’s Dates

The roboticist Hans Moravec, in Mind Children (1988), extrapolated computing power to estimate that supercomputers would match the human brain before 2010. Ray Kurzweil, an inventor of reading machines and synthesizers, made the extrapolation his career: in The Singularity Is Near (2005) he put human-level machine intelligence at 2029 and the Singularity, when people and machines merge, at 2045, and he kept both dates in The Singularity Is Nearer (2024). In 2002 the Lotus founder Mitch Kapor bet him $20,000 on the Long Bets site that “by 2029 no computer, or ‘machine intelligence’, will have passed the Turing Test,” as judged by three human judges in two-hour interviews against three human foils.

From Singularity to Risk

The forecasters were mostly optimists. The next generation took Good’s proviso about docility seriously. Eliezer Yudkowsky founded the Singularity Institute for Artificial Intelligence in 2000, with money from Brian and Sabine Atkins, to accelerate the arrival of AI; he soon decided the danger lay in building it before anyone knew how to make its goals safe, and the institute turned to that problem. It ran the Singularity Summits from 2006 with money from Peter Thiel, sold the name and the summit to Kurzweil’s Singularity University, and in January 2013 became the Machine Intelligence Research Institute. Jaan Tallinn, a Skype co-founder, read Yudkowsky’s essays, met him in March 2009, and became one of the field’s main private backers: he co-founded Cambridge’s Centre for the Study of Existential Risk in 2012, gave to MIRI, and in 2021 led the first funding round of Anthropic.

Nick Bostrom founded the Future of Humanity Institute at Oxford in 2005. His book Superintelligence: Paths, Dangers, Strategies (2014) gave the risk argument an academic form: a superintelligent system would pursue whatever objective it had, and almost any objective, pursued competently enough, gives it reasons to acquire resources and resist being switched off. Elon Musk was among the institute’s largest funders. Oxford’s philosophy faculty froze its fundraising and hiring in 2020 and did not renew its staff contracts, and the institute closed on 16 April 2024.

The Letters of 2023

ChatGPT, released in November 2022, made the old argument urgent to people who had not read it (see The LLM Race). In March 2023 the Future of Life Institute published an open letter asking “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4”; it collected more than 30,000 signatures, including Yoshua Bengio, Stuart Russell, Elon Musk and Steve Wozniak, and no laboratory paused. In May Geoffrey Hinton left Google so that he could talk about the risks. On 30 May 2023 the Center for AI Safety published a statement of one sentence: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Its signatories included Hinton and Bengio, Sam Altman of OpenAI, Demis Hassabis of Google DeepMind and Dario Amodei of Anthropic, the people running the laboratories whose products the statement was about. MIRI announced in 2023–24 that it would work mainly on persuading governments to halt progress toward smarter-than-human AI, and in 2025 Yudkowsky and Nate Soares published the book If Anyone Builds It, Everyone Dies.

Dead End: The Dated Prophecy

Good’s twentieth century ended without an ultraintelligent machine, and so did Moravec’s 2010. Vinge’s thirty years ran out in 2023, the year the extinction statement was signed; the outer limit of his range, 2030, is four years away. Kurzweil’s 2029 has not arrived. What has arrived shows why the dates are hard to score. In March 2025 Cameron Jones and Benjamin Bergen of UC San Diego reported that GPT-4.5, told to adopt a humanlike persona, was judged to be the human 73 percent of the time in a three-party Turing test, more often than the real humans it was compared with. The conversations lasted five minutes; the Kapor–Kurzweil bet specifies two hours with each candidate, and it has not been settled.

The critics have attacked the method as well as the dates. The Microsoft co-founder Paul Allen and Mark Greaves argued in 2011 that science meets a “complexity brake”: “the more we learn, the more we realize there is to know,” so understanding the brain slows down as it proceeds rather than speeding up. Kapor called Kurzweil’s version of the Singularity “intelligent design for the IQ 140 people.” The risk writers replied that their argument does not depend on the date.

📚 Sources