Eliezer Yudkowsky

Abstract

Eliezer Yudkowsky (born 1979) never attended high school or college, co-founded an institute in 2000 to build a superintelligent AI, and spent the next quarter-century arguing that nobody should build one. Through the Machine Intelligence Research Institute, the blog posts later collected as Rationality: From AI to Zombies, the community site LessWrong and a 660,000-word Harry Potter fan fiction, he created the vocabulary of AI risk and the “rationalist” community that carried it into effective altruism, OpenAI and Anthropic. In 2023 he called in Time for a worldwide halt to large AI training runs; in 2025 his book If Anyone Builds It, Everyone Dies became a New York Times bestseller.

From Singularitarian to Alarmist

Yudkowsky was born on 11 September 1979 and was educated at home and by himself; he did not attend high school or university. As a teenager he joined the Extropians mailing list, the 1990s meeting point of transhumanists, cryonicists and early cypherpunks (see Silicon Valley Philosophies), and wrote online essays on how to build a mind smarter than a human one.

In 2000 the internet entrepreneurs Brian and Sabine Atkins, who had read those essays, funded him to found the Singularity Institute for Artificial Intelligence. Its original purpose was to bring the Singularity closer. Within a few years Yudkowsky had concluded that a superintelligence built before anyone knew how to give it the right goals would not care about humans at all, and the institute turned from building AI to the problem of making it safe. His book-length document Creating Friendly AI (2001) gave the problem its first name: “Friendly AI.”

He tried to show the danger with a game. In the AI-box experiment of 2002, Yudkowsky played an AI confined to a text chat, and another person played the gatekeeper who had to keep it locked up. He ran the game twice, in March and July 2002, and reported that both gatekeepers let him out. The transcripts were never published; the point, he said, was that a human jailer is not a safe containment strategy against something smarter than the jailer.

The Sequences and LessWrong

From 2006 Yudkowsky wrote for Overcoming Bias, a group blog with the economist Robin Hanson, and in 2009 he founded LessWrong as a community site under the Singularity Institute. Between 2006 and 2009 he wrote several hundred posts on probability, cognitive bias, philosophy of mind, decision theory and the AI problem. The readers called them “the Sequences.” MIRI collected 333 of them in 2015 as Rationality: From AI to Zombies, a six-book ebook sold on a pay-what-you-want basis.

His widest audience came through fiction. Harry Potter and the Methods of Rationality ran online from 28 February 2010 to 14 March 2015: 122 chapters and more than 660,000 words in which an orphan raised by an Oxford biochemist enters Hogwarts and applies the scientific method to magic. Each chapter linked back to the ideas of the Sequences, and it became one of the main routes to LessWrong: in the site’s 2012 reader survey, 22 percent of respondents said the story had brought them there.

The Peter Thiel-funded Singularity Summits ran from 2006. In December 2012 the institute sold its name and the summit to Singularity University and in January 2013 renamed itself the Machine Intelligence Research Institute (MIRI), based in Berkeley. Jaan Tallinn read Yudkowsky’s essays, met him in 2009 and became one of the field’s main funders.

Influence on the Labs

The community Yudkowsky built supplied a large share of the people, money and arguments behind the AI-safety movement. LessWrong merged with the Oxford-based effective altruism movement around the idea that reducing extinction risk outweighs other causes. Nick Bostrom, who had also come through the Extropians list, gave the argument academic form in Superintelligence (2014). Elon Musk recommended the book in August 2014 with the warning that AI was “potentially more dangerous than nukes,” and co-founded OpenAI the next year.

Yudkowsky regarded the labs that grew out of these ideas as making things worse. OpenAI and Anthropic argued that the safest course was for careful people to build frontier models first; Yudkowsky argued that nobody knew how to align such systems and that a race to build them was a race to the end.

“Shut It All Down”

After ChatGPT his position reached a mass audience. In the Bankless podcast episode “We’re All Gonna Die” (February 2023) he told two crypto hosts, who had expected to talk about AI and markets, that he expected humanity to lose. On 29 March 2023, a week after the Future of Life Institute asked for a six-month pause on large training runs, he wrote in Time that a pause was not enough. He asked for an indefinite, worldwide moratorium on large training runs, with no exceptions for governments or militaries, enforced by international agreement, and wrote that states should be willing to “destroy a rogue datacenter by airstrike.” Time put him on its 2023 list of the 100 most influential people in AI.

MIRI announced in January 2024 that it would work mainly on persuading governments to halt progress toward smarter-than-human AI. With MIRI’s president Nate Soares he published If Anyone Builds It, Everyone Dies in September 2025. The book argues that a superintelligence built with current methods would pursue goals no one chose and would kill everyone as a side effect, and it ends with a call for an international treaty. It entered the New York Times bestseller list.

Dead End: Friendly AI by Proof

MIRI’s original research program aimed to make AI safe by mathematics: to find a decision theory and a goal architecture whose safety could be proven before the system was built. For about fifteen years its researchers worked on logical uncertainty, decision theory and self-referential reasoning. The approach assumed that powerful AI would be designed from understood components. Instead it arrived as large neural networks trained on data, whose internals nobody designed and whose goals nobody wrote down. In January 2024 MIRI wrote that it had grown “more pessimistic that such work will have time to bear fruit” and that “policy and communications will be a higher priority for MIRI than research going forward.” The field it founded continued in other forms, mostly at the labs Yudkowsky opposes: interpretability, red-teaming and training-time techniques, applied to systems built in exactly the way his program had hoped to avoid.