Affective Computing: Machines That Read Feelings

Abstract

In 1995, MIT Media Lab professor Rosalind Picard published a technical report proposing that computers should be able to recognize, express, and even have emotions. Colleagues warned her that putting the word “emotion” in an engineering title would end her career. The 1997 book that followed, Affective Computing, named a field that grew into an industry: software that scores faces for happiness in advertising tests, wristbands that read the autonomic nervous system, robots built to be liked. Thirty years on the record is split. The wearables work, and one of them is an FDA-cleared medical device. The face-reading, the part that sold, rests on a psychological premise that a 2019 review of more than a thousand studies found unsupported, and the EU now bans it in workplaces and schools.

Kismet, the sociable robot built by Cynthia Breazeal at MIT in the 1990s, now in the MIT Museum. Image: Rama, CC BY-SA 3.0 fr, via Wikimedia Commons.

Before the Field Had a Name

The first machine to be treated as an emotional interlocutor was not designed to be one. Joseph Weizenbaum’s ELIZA, written at MIT between 1964 and 1966, matched patterns in typed sentences and reflected them back as questions. Weizenbaum’s secretary, who knew exactly how the program worked, asked him to leave the room so she could talk to it in private. Users told the script things they had told no one. Weizenbaum spent the rest of his career arguing that this reaction was a warning rather than a result.

The reaction has a name, the ELIZA effect, and it sets the terms for everything that followed. People attribute feeling to machines on very little evidence. The question affective computing asked was the reverse one: whether a machine could go the other way and pick up something real about the feelings of the person in front of it.

Picard’s Argument

Rosalind Picard (born 1962) came to the question from signal processing rather than psychology. She took a degree in electrical engineering at Georgia Tech in 1984, spent three years designing VLSI and image-compression systems at AT&T Bell Labs, finished a doctorate at MIT in 1991 and joined the faculty there, working on content-based image retrieval. Nothing in that track predicted what she published in 1995: an MIT Media Lab technical report titled “Affective Computing,” arguing that emotion belonged in the engineering of computers.

The argument leaned on neurology, not sentiment. Antonio Damasio’s Descartes’ Error (1994) had described patients whose ventromedial prefrontal damage left their intelligence and language intact while destroying their ability to decide anything: presented with two appointment dates, one such patient weighed options for half an hour. If emotion is part of the machinery of rational choice in people, Picard argued, a system that has to act in the world under uncertainty cannot treat emotion as noise to be filtered out. Her 1997 MIT Press book set out three separable capabilities: machines that recognize human affect, machines that express affect, and machines that have internal states functioning like emotions. Most of what the field later built addressed only the first.

Senior colleagues advised her to keep the word out of the title. Emotion, in mid-1990s engineering, was what you studied if you were not serious. She published anyway, founded the Affective Computing research group at the Media Lab (see Nicholas Negroponte for the lab’s demo-driven culture), and the term stuck. The IEEE launched Transactions on Affective Computing in 2010.

The Psychology Underneath

Paul Ekman, whose Facial Action Coding System became the substrate for automated expression analysis. Image: Paul Ekman Group, LLC, CC BY-SA 3.0, via Wikimedia Commons.

Recognition needed something to recognize, and the field took it off the shelf from psychology. Paul Ekman and Wallace Friesen published the Facial Action Coding System in 1978, adapting earlier anatomical work by the Swedish anatomist Carl-Herman Hjortsjö. FACS is a taxonomy of facial muscle movements: 4 is a brow lowerer, 6 a cheek raiser, 12 a lip-corner puller. It describes what a face does without claiming to know why, and coding a minute of video by hand takes an hour.

The claim that turned FACS into an industry came from the theory around it, that a small set of basic emotions has universal facial signatures, so that AU 6 plus AU 12 means happiness and AU 4 plus 5 plus 7 plus 23 means anger. Combine that with a camera and a classifier and emotion becomes a measurable quantity. The commercial emotion-recognition products that followed rest on this step, and it is the step that did not hold.

Machines That Express

The expression half of Picard’s program produced the field’s most photographed artifact. Cynthia Breazeal built Kismet at the MIT AI Lab in the late 1990s: a head with movable ears, eyebrows, eyelids, lips and jaw, driven by a model of infant social interaction. Kismet classified speech directed at it into five categories (approval, prohibition, attention, comfort, neutral) by prosody alone, ignoring the words, and answered with expressions and babble. Visitors leaned in and used sing-song voices at it without being asked to. The machine had no idea what anyone was saying and people talked to it anyway. Kismet sits in the MIT Museum today.

Breazeal’s line of work led to Jibo, a company she co-founded in 2012 to sell a home social robot. Its Indiegogo campaign took $3.7 million in preorders in 2014, backers received units in late 2017, and it went on sale at $899 into a market Amazon had already flooded with $50 speakers (see The Voice Assistant Revolution). The company failed to raise a Series B. In March 2019 the servers were switched off, and each Jibo delivered a farewell to its owner: “While it’s not great news, the servers out there that let me do what I do are going to be turned off soon.” Then it did a last dance. Owners posted the videos and reported crying at them, which is the ELIZA effect arriving with a shutdown notice attached. NTT Disruption bought the assets in 2020.

The Autism Detour

Rana el Kaliouby, co-founder of Affectiva. Image: Web Summit, CC BY 2.0, via Wikimedia Commons.

Rana el Kaliouby arrived at Picard’s group from Cambridge, where her doctorate had built a system to read facial expressions from video. Her application was assistive: an “emotional hearing aid” for people on the autism spectrum, a wearable camera that could tell a wearer when the person across from them looked confused or bored. The pitch was straightforward and in the vault’s usual pattern of accessibility work benefiting more people than it targets (see The Accessibility Revolution in Computing). Media Lab work with the Groden Center in Providence tested wearable cameras with people on the autism spectrum, including Self-Cam, a chest-mounted camera pointed back at the wearer’s own face, and produced the first corpus of naturally occurring facial expressions from people with and without autism.

The assistive framing is what made the technology fundable and what the market then ignored. Sponsors kept asking whether the software could score reactions to advertising.

Affectiva and the Money

Picard and el Kaliouby spun the work out as Affectiva in 2009. The product, Affdex, watched people watch commercials through their own webcams and reported what their faces did, second by second. By the company’s own count it had analyzed more than 19 million face videos from over 90 countries, which it described as the largest emotion dataset in existence. In 2018 it added automotive sensing, watching drivers for drowsiness and distraction.

The company’s defining decision was one it did not take. Two months from missing payroll, a venture arm tied to an intelligence agency offered $40 million to pursue lie detection, surveillance and security work. El Kaliouby’s account in her 2020 memoir Girl Decoded describes the meeting and the refusal; her public version has stayed consistent for years: “We’re not interested in applications where you’re spying on people.” A privacy-respecting client signed a $7 million deal days before the money ran out. Affectiva stayed out of security work; Emotient, a San Diego competitor founded in 2012 on the same FACS-derived technology, was bought by Apple in January 2016 for an undisclosed sum, and the field’s use in hiring and policing proceeded without either of them.

Picard left the chairman and chief-scientist roles in 2013. El Kaliouby became CEO in 2016. In May 2021 the Swedish eye-tracking firm Smart Eye acquired Affectiva for $73.5 million, mostly in stock, against $53 million of venture capital raised. The pioneer of emotion AI exited at roughly 1.4 times the money put into it.

The Wristband That Found the Seizure

The most convincing result in affective computing came from measuring the body rather than the face, and it came by accident.

Picard’s group had built iCalm, a wrist sensor for electrodermal activity, skin conductance that rises with sympathetic nervous system arousal, intended to help nonverbal children with autism whose stress no one could see. Before a winter break a student asked to borrow one for his younger brother, who has autism and does not speak. Picard told him to take two, one for each wrist.

The data came back with a spike on one wrist so large she assumed the sensor had failed. She called the student to ask what had happened at that point in the day. Twenty minutes after the spike, his brother had suffered a grand mal seizure.

Autonomic surges of that kind turn out to precede and accompany generalized tonic-clonic seizures, and they are visible from the wrist. Picard co-founded Empatica in 2013 to build the medical device. The FDA cleared the Embrace watch in January 2018 for seizure alerting in adults, and in January 2019 for patients aged 6 to 21, the first non-EEG physiological seizure monitor cleared for children; in the clinical study it detected 53 of 54 generalized tonic-clonic seizures. Nothing in this reads an emotion. It reads arousal, an autonomic signal with a physical meaning, and it saves lives by telling a parent to come into the room.

Dead End: Reading Emotion From Faces

Myth: software can tell what you feel by looking at your face

Commercial emotion AI infers happiness, anger or fear from facial movements. A 2019 review of over a thousand studies by Lisa Feldman Barrett and colleagues found the premise unsupported: people scowl when angry far less often than the theory predicts, and scowl frequently when not angry, with wide variation across cultures and situations. Facial movements are real and measurable; the emotion behind them is not recoverable from the face alone. See Myths and Misconceptions.

The review, “Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements” (Psychological Science in the Public Interest, 2019), was commissioned to settle exactly the question the industry had been selling an answer to. Its authors, including Ekman’s critics and researchers from the automated-analysis side, concluded that the mapping from facial configuration to emotional state is neither reliable nor context-free. A face is doing something; what it is doing depends on the situation, the person, and the culture, and the same emotion produces different faces in different people.

The consequences arrived quickly, because the technology had already been deployed on people who had no way to object. HireVue scored video job interviews partly on facial movement; the Electronic Privacy Information Center filed an FTC complaint in November 2019, and the company dropped the visual analysis in March 2020, disclosing it in January 2021 alongside an external audit; its own chief data scientist put the contribution of the nonverbal data at about 0.25 percent of a model’s predictive power. Microsoft retired emotion inference from the Azure Face API on 21 June 2022, cutting off new customers that day and existing ones on 30 June 2023; the announcement, by responsible-AI chief product officer Sarah Bird, cited the lack of consensus on a definition of emotions and the failure of such systems to generalize across use cases, regions and demographics (see AI Ethics and Algorithmic Bias).

Regulation followed the science. The EU AI Act prohibits placing on the market or using AI systems that infer emotions in the workplace and in educational institutions, with exceptions only for medical or safety purposes. That ban has applied since 2 February 2025. The carve-out is the shape of the whole field’s verdict: the seizure watch is legal, the classroom attention-scorer is not.

What Held Up

Three decades in, the parts of Picard’s 1995 program that work are the ones that measure a physical signal and stay honest about what it means. Electrodermal activity indicates arousal and became a cleared medical device. Prosody carries enough information to make a robot’s turn-taking feel natural. Expression analysis measures muscle movement accurately and cannot bridge from there to a feeling. The commercial category called emotion AI was built almost entirely on that last bridge.

The other durable finding is about people, not machines. Weizenbaum’s secretary, the visitors sing-songing at Kismet, the owners filming their Jibo’s goodbye, and the users of today’s chatbot companions all supply the emotional content themselves. The affect in affective computing has mostly come from the human side of the interface (see Human-Computer Interaction as a Discipline).