Skip to content

The Natural Language Processing Revolution

Abstract

From rule-based chatbots to neural networks that handle language at human level, the history of computational linguistics is a history of classical AI failing and statistical methods winning. The Transformer (2017) ended decades of dispute and opened the era of large language models.

ELIZA and the Illusion of Understanding

In 1966, Joseph Weizenbaum at MIT published a program called ELIZA, named after Eliza Doolittle from Shaw’s Pygmalion. It simulated a psychotherapist through simple pattern matching: “My mother hates me” became “Tell me more about your mother.” No meaning, just pattern rules.

What shocked Weizenbaum: people knew they were talking to a computer and became emotional anyway. His own secretary asked him to leave the room so she could speak privately with ELIZA. The phenomenon became known as the ELIZA effect: the human tendency to ascribe understanding to computers that have none.

ELIZA was not a breakthrough. It was a warning. Weizenbaum himself became the sharpest critic of AI research; his book Computer Power and Human Reason (1976) is a reckoning with the claim that machines could ever truly understand.

The Symbolic Era: Grammars, Rules, Ontologies

AI research from the 1960s to the 1990s believed in language as a formal system. If you could fully formalize the grammar of a language (context-free grammars, Chomsky’s transformational grammars, semantic networks), understanding would follow.

The result was laborious and fragile:

  • SHRDLU (Terry Winograd, MIT, 1970): a system that understood English, but only in a world of colored blocks on a virtual table. Outside this microworld it failed completely.
  • Cyc (Doug Lenat, 1984–present): the most ambitious symbolic project in history. Goal: encode all human common-sense knowledge in an ontology. Decades of manual work, millions of facts, and still no system that truly understands natural language.
  • Hand-coded parsers: industrial NLP systems of the 1980s/90s used handwritten grammar rules. Maintenance effort: enormous. Coverage: limited. Exceptions: endless.

The problem: natural language is not a formal language. It is ambiguous, context-dependent, culturally shaped, and constantly changing. Every rule has exceptions. Every exception has exceptions.

The Statistical Turn

The shift came in the late 1980s and 1990s, unnoticed at first. IBM researcher Frederick Jelinek and his group developed statistical models for automatic speech recognition that were based not on grammar rules but on probabilities: which word follows which?

Jelinek’s famous line: “Every time I fire a linguist, the performance of our speech recognizer improves.” A provocation, but also an empirical observation.

Statistical machine learning successively took over every NLP subfield:

  • Hidden Markov Models (HMMs) for part-of-speech tagging
  • Naive Bayes for spam filtering (Paul Graham popularized the concept in 2002)
  • Maximum entropy models and Conditional Random Fields (CRFs) for named entity recognition

No system “understood” language. But statistical systems were robust, scalable, and (more important) measurably improved by more data.

Word2Vec: Meaning as a Vector

In 2013, Tomas Mikolov and colleagues at Google published a paper that electrified the NLP world: Word2Vec.

The idea: train a neural network to predict a word from its context (or vice versa). The network must learn to represent words in a high-dimensional space, and in doing so encode semantic similarity.

The result: vectors you can do arithmetic with.

King − Man + Woman ≈ Queen
Paris − France + Germany ≈ Berlin

Meaning was suddenly algebraic. Linguistic analogies could be expressed as vector operations. That was not a trick; it was fundamental evidence that neural networks can extract semantic structure from raw text.

Words with similar meanings sat close together in the vector space. Synonyms clustered. Antonyms did not. Country names and capital names formed parallel structures.

GloVe (Pennington et al., Stanford, 2014) refined the method. The paradigm was set: distributed representations: meaning as vector, not as symbol.

The Transformer: Attention Is All You Need

On June 12, 2017, eight Google Brain researchers (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin) published a paper with the self-confident title “Attention Is All You Need”.

It was an architectural decision that changed everything.

Abstract

The attention mechanism allows a model, when processing one word, to weight all other words of the sentence simultaneously, instead of working sequentially forward or backward. Self-attention answers, for each token: which other tokens are relevant to me, and how strongly?

The elegance: attention is differentiable, parallelizable, and scales naturally to long contexts. RNNs had to “pass along” information through sequential time steps, a bottleneck for long dependencies. Transformers process all positions at once. That made training efficiently scalable on modern GPUs for the first time, and laid the foundation for every LLM that followed.

Before, recurrent neural networks (RNNs) and their descendant, the Long Short-Term Memory networks (LSTMs) (Hochreiter & Schmidhuber, 1997), dominated. They processed sequences step by step, which made long dependencies hard to learn and prevented massively parallel training.

The Transformer replaced recurrence entirely with attention. No sequential state. Just matrices and probabilities.

BERT, GPT, and the Road to the LLM Era

BERT (Google, October 2018)

Bidirectional Encoder Representations from Transformers: Jacob Devlin and colleagues at Google trained a Transformer model on two tasks: masked language modeling (predicting randomly masked words) and next sentence prediction.

The result: a pretrained model that could be fine-tuned for practically any NLP task. BERT reached state of the art on eleven standard benchmarks, some by a wide margin. It was the first proof that transfer learning is as powerful in NLP as in computer vision (ImageNet models for image recognition).

GPT and GPT-2 (OpenAI, 2018–2019)

OpenAI pursued a different direction: instead of a bidirectional encoder, an autoregressive decoder. The model learns to predict the next token. GPT (Generative Pre-trained Transformer, June 2018) showed the same transfer learning potential.

GPT-2 (February 2019) was the first moment of public attention. OpenAI initially withheld the full model, citing fear of misuse; the press condensed the decision to “too dangerous to release.” Reactions were split: some saw a real threat, others a PR strategy.

GPT-2 could generate text that astonished people. It was not understanding, but it was coherent, contextual writing at a scale nobody had seen before. The direction was clear: more parameters, more data, more compute.

Dead End: Symbolic NLP Systems

Warning

The fall of classical computational linguistics

Decades of work on hand-coded grammars, parse trees, semantic networks, and ontologies like Cyc or WordNet ended not by refutation but by being overtaken. Statistical methods and neural networks beat symbolic systems on nearly all benchmarks, without ever offering the elegant explanatory power of formal grammars.

The paradox: symbolic systems are interpretable; you can trace why a decision was made. Neural networks are black boxes. The black box won anyway, because it works better.

What remained: rule-based systems in niches where explainability or legal compliance matter more than raw accuracy (medical decision systems, legal document analysis, critical infrastructure). And Chomsky’s transformational grammar as a linguistic theory, but no longer as an engineering approach for NLP products.

Legacy

The NLP revolution is a story about epistemology: what does “understanding” mean? ELIZA effects fooled people. Symbolic systems formalized language without grasping it. Statistical models learned patterns without meaning. Transformers learned patterns at a level that simulates meaning, and the boundary became blurry.

The practical consequences came fast: Google Translate improved abruptly through neural networks (2016). Voice assistants got better. Spell checking, sentiment analysis, automatic summarization: everything benefited.

And then came GPT-3 (2020) with 175 billion parameters. ChatGPT (2022). The LLM era. That was no longer NLP as a tool. That was NLP as infrastructure, as the universal interface between human and machine.

Weizenbaum would have seen it coming: the question was never whether machines can think. The question was whether people would stop asking.

📚 Sources