Human-Computer Interaction: From Terminals to Touch

Abstract

Human-Computer Interaction (HCI) emerged as a formal academic discipline in the early 1980s at the intersection of computer science, cognitive psychology, and industrial design. It asked a question the computing industry had largely ignored: what makes a computer usable by a human being? The answer, developed over four decades by researchers at Carnegie Mellon, Bell Labs, Xerox PARC, and Apple, transformed computing from a specialist tool into an everyday artifact, and in doing so, quietly reshaped every screen that billions of people now touch dozens of times a day.

Before HCI: The User as an Afterthought

For the first three decades of computing, the human being at the terminal was not a design consideration. The IBM 704 demanded that users feed it punched cards in a precisely specified format; the machine’s convenience was paramount, the user’s an afterthought. Time-sharing systems of the 1960s gave users interactive access, but “interactive” meant a blinking cursor and a cryptic command language that encoded the machine’s internal logic rather than the user’s intentions. Systems spoke to their operators in the language of interrupts, registers, and file descriptors. Learning to use a computer meant learning to think like one.

Douglas Engelbart at the Stanford Research Institute was the first major figure to invert this assumption. His NLS (oN-Line System), demonstrated in December 1968 in what became known as “The Mother of All Demos,” showed a complete vision of human-augmented computing: a mouse for pointing, hypertext links for navigation, video conferencing, and collaborative editing, all in service of making computers extend human intellectual capability rather than demanding human adaptation to machine logic. Engelbart’s fundamental question (how can computers augment human intellect?) would eventually become HCI’s founding question.

Alan Kay at Xerox PARC pushed further with his Dynabook concept (1972): a personal computer the size of a notebook, with a graphical interface, designed to be usable by children. Kay’s thinking incorporated the developmental psychology of Jean Piaget and the educational philosophy of Seymour Papert, computers as tools for learning and thinking, not just calculating. The Alto, which Xerox PARC built starting in 1973, embodied many of these ideas in hardware: a bitmap display, a mouse, overlapping windows, and icons. It was a prototype that most people never touched and a vision that everyone in the industry would eventually copy.

Fitts’ Law and the Mathematics of Pointing

The scientific foundation of HCI was laid before the field had a name.

In 1954, Paul Fitts, an experimental psychologist working on human performance in aircraft cockpit design, published a study in the Journal of Experimental Psychology that quantified the time required for a human to point accurately at a target. His finding (now known as Fitts’ Law) states that the time to move a pointer to a target is a function of the distance to the target and the width of the target:

T = a + b × log₂(2D/W)

where T is movement time, D is distance, W is target width, and a and b are empirically determined constants.

Fitts derived this from studies of aircraft control movements. He was interested in aviation ergonomics, not software design. But the formula precisely predicts the time it takes to move a mouse cursor to a menu item, click a button, or select text, any pointing task on any display. When Xerox PARC researchers in the 1970s began studying how people interacted with the Alto’s screen and mouse, Fitts’ Law became an essential tool for evaluating interface designs. Making a button larger, or placing frequently used controls near where the cursor is likely to be, has a quantifiable effect on user performance. HCI, at its foundation, is partly the application of 1954 experimental psychology to software design.

The Birth of a Discipline, 1983

The formal establishment of HCI as a discipline is conventionally dated to 1983 and to one book.

Stuart Card, Thomas Moran, and Allen Newell published The Psychology of Human-Computer Interaction in 1983. Card and Moran worked at Xerox PARC; Newell was at Carnegie Mellon, where he had supervised Moran’s doctorate. At PARC the three developed the GOMS model (Goals, Operators, Methods, Selection rules) the first formal analytical framework for predicting human performance with computing interfaces. GOMS decomposed a task into a hierarchy of goals (write a document), methods (use a word processor), operators (press a key, move a mouse), and selection rules (which method to apply when). By quantifying the time and error rates of individual operators and summing them, GOMS could predict how long it would take an expert user to complete a task with a given interface design, before building the interface. It was engineering applied to human behavior, and it made HCI a science rather than intuition.

Newell, one of the founders of artificial intelligence and cognitive psychology, brought broader ambition. His concept of the cognitive architecture (a unified theory of how the human mind processes information) framed HCI’s theoretical project: build interfaces that match the capabilities and limitations of human cognition, rather than demanding that humans overcome those limitations through training.

The same year, the first CHI conference (Conference on Human Factors in Computing Systems) was held in Boston, organized by the ACM’s Special Interest Group on Computer-Human Interaction (SIGCHI). It attracted over 1,000 attendees, a strong debut for a new field, building on a 1982 precursor gathering in Gaithersburg, Maryland. By 2019, CHI had become one of the largest academic computing conferences in the world: the Glasgow edition took 2,958 full-paper submissions and accepted 703, out of 4,652 submissions across all tracks. HCI had not just become a discipline; it had become one of computing’s most active research areas.

Ben Shneiderman and Direct Manipulation

The same year, Ben Shneiderman founded the Human-Computer Interaction Laboratory (HCIL) at the University of Maryland, which he directed until 2000. In 1982 he had coined the term direct manipulation for the interaction style the GUI pioneers had arrived at by instinct: continuous visual representation of the objects of interest, physical actions in place of typed command syntax, and immediate, reversible, visible feedback. The definition turned a designer’s preference into a testable claim, and Shneiderman’s experiments supplied evidence for why dragging a file to a trash can beats typing a delete command. One of the pioneers who reached the style by instinct was Larry Tesler at Xerox PARC, whose modeless Gypsy editor (1974–75) removed the command/insert distinction and introduced cut, copy, and paste; his rule that complexity can only be moved between the software and the user, never eliminated, is now taught as the Law of Conservation of Complexity.

His laboratory produced two artifacts with reach far beyond academia. Hyperties, a hypertext browsing system built in the mid-1980s, used highlighted selectable text as its links and was used to produce the first electronic scientific journal issue, a hypertext edition of the July 1988 Communications of the ACM. Shneiderman’s account is that this work supplied the “hot spots” behind the links in Tim Berners-Lee’s 1989 proposal for the World Wide Web; either way, the highlighted selectable link predates the Web by several years. The second artifact is the treemap (1991), a space-filling visualization of hierarchical data now standard in disk-usage tools, stock-market dashboards, and newsroom graphics.

Shneiderman also wrote one of the field’s standard textbooks, Designing the User Interface (1986, now in its sixth edition), with its “Eight Golden Rules” of interface design, and compressed information visualization into a single sentence in his 1996 paper “The Eyes Have It”: “Overview first, zoom and filter, then details-on-demand.” Since the field’s earliest meetings he photographed his colleagues at conferences; the collection appeared in 2019 as Encounters with HCI Pioneers, a photo journal of sixty researchers that doubles as the discipline’s family album.

Don Norman and the Design of Everyday Things

If Card, Moran, and Newell gave HCI its scientific machinery, Don Norman gave it its cultural influence.

The field also produced instructive failures. The most studied is the anthropomorphic assistant, which assumed users wanted a personality rather than a predictable tool: see Dead End: Microsoft Bob and Clippy.

Norman arrived at UCSD’s Cognitive Science department after a career in experimental psychology and a fellowship at the Center for Advanced Study in the Behavioral Sciences. His 1988 book The Psychology of Everyday Things (reissued in 1990 as The Design of Everyday Things) was addressed not to programmers but to anyone who had ever struggled to operate a door, a stove, or a telephone. Its argument was systematic: when you fail to use an object correctly, the object is usually at fault. Good design provides:

Affordances: visual cues that indicate how an object is to be used (a button affords pressing; a handle affords pulling).
Visibility: the state of the system should be apparent from looking at it.
Feedback: actions should produce perceptible results.
Mental models: the user’s conceptual model of how a system works should match its actual behavior.
Constraints: design should prevent users from making errors, not merely warn them after.

The book has stayed in print for nearly forty years across two retitled editions and entered the working vocabulary of product designers, interface engineers, and business managers. Norman’s concept of the affordance (borrowed from ecological psychologist James Gibson and adapted for design) became perhaps the most widely used term in interface design discourse. His critique of “Norman doors” (doors with handles you push) became a standard example in design education worldwide.

Norman joined Apple in 1993 as Vice President of the Advanced Technology Group, one of the first times a cognitive psychologist had held a senior role in a major technology company. The move signaled a shift: HCI was no longer only an academic discipline.

Jakob Nielsen and the Engineering of Usability

While Norman addressed broad design principles, Jakob Nielsen developed practical engineering methods for improving interfaces systematically.

Nielsen and Rolf Molich introduced heuristic evaluation at CHI in 1990. Nielsen then refined the list: for his 1994 paper “Enhancing the Explanatory Power of Usability Heuristics” he took a database of 249 documented usability problems, ran a factor analysis over a large set of candidate principles, and kept the subset that explained the most problems. The result is the canonical ten heuristics:

Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from errors
Help and documentation

These heuristics enabled heuristic evaluation: a usability inspection method in which evaluators (even non-users) assess an interface against the list and identify violations. Nielsen demonstrated that five evaluators using the heuristics could identify 75% of usability problems in an interface, making systematic usability testing economically practical for the first time.

Nielsen went on to establish the Nielsen Norman Group with Don Norman in 1998, became the most widely cited HCI researcher in the practitioner community, and published analyses of web usability that shaped how the early web was designed. His finding that users read web pages in an F-pattern (scanning horizontally across the top, then down the left margin) fundamentally influenced web layout conventions that persist to the present day.

From Usability to User Experience

By the late 1990s usability had a formal definition: ISO 9241-11 (1998) described it as the extent to which specified users can achieve specified goals with effectiveness, efficiency, and satisfaction. For most practitioners that definition was the whole job. Fun, beauty, and pleasure were considered out of scope, dismissed as an esoteric variant of “satisfaction” or as unmeasurable. A dissenting voice came early: at the 1997 German Software-Ergonomie conference in Dresden, Sun Microsystems’ Bob Glass told the audience, “If you’re still talking about ease of use then you’re behind. It is all about the joy of use.”

The systematic push beyond usability came in part from Germany. Marc Hassenzahl and Michael Burmester, then working in a Siemens user interface design group, kept meeting products their methods could not evaluate. When Siemens Medical redesigned its Syngo medical imaging interface in 1998–99, classic usability tests flagged the unfamiliar widgets as problems while saying nothing about the interface’s elegance or novelty (the design later became Siemens’ standard). Together with Franz Koller, cofounder of the Ludwigsburg consultancy User Interface Design (UID), they published “Usability ist nicht alles” (“Usability isn’t everything”) in the first issue of the journal i-com in 2002. Their instrument was the AttrakDiff questionnaire, which separates pragmatic quality (a product’s perceived ability to support task completion) from hedonic quality (its perceived ability to satisfy self-oriented needs such as stimulation or self-expression). Early studies showed both qualities contribute about equally to a product’s appeal.

The field resisted for a while; ISO 9241-11 defenders argued that enjoyment was already covered under “satisfaction.” The turn became respectable in 2004 when Don Norman published Emotional Design: Why We Love (or Hate) Everyday Things, reversing his own cognition-first stance from The Design of Everyday Things. In 2010 the concept entered the standards: ISO 9241-210 defined user experience as a “person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service.” Twenty years after their provocation, Hassenzahl, Burmester, and Koller published a retrospective in the same journal, this time titled “User Experience Is All There Is.”

The GUI Timeline: From Alto to iPhone

The graphical user interface evolved through a series of landmarks, each building on its predecessors:

Xerox Alto (1973): the prototype that established the paradigm, bitmap display, mouse, overlapping windows, icons. Never sold commercially.

Xerox Star (1981): the first commercial GUI workstation, at $16,000. Beautiful, influential, commercially unsuccessful.

Apple Lisa (January 1983): Apple’s first GUI machine, at $9,995. Too expensive; estimates of total sales over its two years on the market range from 10,000 to 60,000 units.

Apple Macintosh (1984): at $2,495 with the “1984” Super Bowl advertisement. The GUI made accessible. Steve Jobs had seen the Alto at Xerox PARC in 1979 and understood what it meant in a way Xerox’s management had not.

Windows 1.0 (1985): Microsoft’s first tiled-window interface for DOS. Crude but important as a statement of direction.

Windows 3.0 (1990): the first commercially successful Windows, establishing the Microsoft GUI paradigm that would dominate for two decades.

Touchscreens arrived on a different timeline. Finger-based touch input was researched at CERN and the University of Kentucky in the 1970s, where the first finger-driven touch screens were demonstrated for industrial control applications. Multi-touch predates its popularizer by more than two decades: Nimish Mehta built the first multi-touch input system at the University of Toronto in 1982, and Toronto’s Input Research Group around Bill Buxton demonstrated a capacitive multi-touch tablet in 1984, publishing it at CHI in 1985. Buxton, later a principal researcher at Microsoft Research, has maintained a public chronology of this prehistory as a corrective to the belief that multi-touch began in Cupertino. Resistive touchscreens appeared in consumer devices through the 1990s. But touch remained niche until Apple’s multi-touch iPhone in January 2007, a device with no stylus, no physical keyboard, and no hardware buttons for applications. The entire interaction model was finger gestures on glass.

The iPhone’s interface established new HCI conventions as rapidly as the Macintosh had in 1984: swipe, pinch, tap, double-tap. Within five years, these gestures were so universal that toddlers attempted them on printed photographs.

Tangible Interfaces

Not everyone accepted glass as the endpoint. Hiroshi Ishii, who joined the MIT Media Lab in 1995 and founded its Tangible Media Group, argued in the 1997 CHI paper “Tangible Bits” (with Brygg Ullmer) that pixels behind glass waste the human talent for grasping, arranging, and feeling physical things. His group built interfaces where digital information takes graspable form: physical tokens that stand for data, surfaces that respond to objects placed on them, ambient displays at the edge of attention. In 2012 he extended the program to “Radical Atoms,” materials that would change shape under computational control. Most of it remains laboratory work, but its traces are visible wherever an interface reaches beyond the flat screen.

Down the hall, Rosalind Picard’s group pushed on a different limit of the desktop model: an interface that registers what the user is typing but nothing about the state they are in. Her 1997 book Affective Computing argued for machines that sense and express emotion, a programme that produced a cleared medical wearable and a face-reading industry built on psychology that did not survive review (see Affective Computing).

Teaching the Field

HCI spread as much through books and classrooms as through products. The graphics layer beneath every interface was taught from James Foley and Andries van Dam’s Fundamentals of Interactive Computer Graphics (1982), expanded in 1990 (with Steven Feiner and John Hughes) into Computer Graphics: Principles and Practice, known as the bible of computer graphics for both its authority and its weight. Van Dam had built the first hypertext system with Ted Nelson at Brown in 1967 (see Ted Nelson and Hypertext).

John M. Carroll, who founded the User Interface Institute at IBM’s Watson Research Center in 1984, attacked the training problem from the documentation side. His book The Nurnberg Funnel (1990) is named for the legendary funnel that pours knowledge directly into a pupil’s head, the device his data showed does not exist: people learn software by acting, erring, and recovering, not by reading systematic manuals. The resulting doctrine of minimalist instruction made short, task-first documentation the industry norm.

Randy Pausch at Carnegie Mellon built Alice, a 3D programming environment in which novices learn to program by animating virtual worlds. His lecture “Really Achieving Your Childhood Dreams” (September 18, 2007), delivered while he was terminally ill with pancreatic cancer, spread across the early video web; the book version, The Last Lecture, passed 4.5 million copies in the United States and was translated into 46 languages. Pausch died on July 25, 2008, at 47.

Skeuomorphism vs. Flat Design: The Aesthetics War

The period 2007–2013 produced an unexpected HCI debate with genuine design consequences: the contest between skeuomorphism and flat design.

Skeuomorphism (using visual metaphors borrowed from physical objects) had been a principle of Apple’s interface design under Steve Jobs and Scott Forstall. The iOS Notes app used a yellow legal pad texture; the Bookshelf app used wooden shelves; the calendar used stitched leather. The argument was that familiar textures helped users understand unfamiliar interactions by mapping them onto physical objects they already knew.

Flat design, favored by Microsoft’s Metro design language (2010) and promoted by designers who argued skeuomorphism was visual noise, stripped away textures and gradients in favor of pure color, typography, and whitespace. The argument was that once users understood touch interfaces, realistic textures were unnecessary decoration that added visual complexity without aiding comprehension.

Apple resolved the debate institutionally: when Jony Ive took over iOS design from Forstall in 2012, iOS 7 (2013) replaced Apple’s skeuomorphic aesthetic with a flat, translucent design language. The shift was jarring for existing users and influential across the industry. Android and web design followed. The skeuomorphic era ended not because the argument was settled theoretically, but because Apple chose a side.

Accessibility as HCI

HCI’s most consequential and least celebrated achievement may be its work on accessibility, the effort to make computing usable by people with visual, motor, hearing, and cognitive disabilities.

Screen readers (software that converts text to speech for blind and low-vision users) developed through the 1980s alongside the rise of personal computing. IBM researchers Jim Thatcher and Jesse Wright built one of the first screen readers for DOS, first called PC-SAID and released in 1984 as IBM Screen Reader, a name that became the generic term for the whole category. Thatcher later led IBM Screen Reader/2, the first screen reader for a graphical interface. JAWS (Job Access With Speech), released for DOS in 1989, became the most widely used and remains so. Their existence depends on the same metadata structures (accessible names, roles, states) that HCI researchers developed for structured document design.

Voice control, first developed to assist users with motor disabilities who could not use a keyboard or mouse, became mainstream technology through products like Apple’s Siri (2011) and Amazon’s Alexa (2014). The accessibility research that motivated voice control for people with ALS or spinal cord injuries produced the voice interfaces that billions of people now use by preference.

Switch access (controlling a computer with a single button) was developed for users with severe motor disabilities. The same principles informed the design of game controllers, which reached users who needed simplified input for different reasons.

HCI’s identity as a discipline positioned at the intersection of computer science, cognitive psychology, and industrial design meant it was structurally equipped to ask questions about human diversity that pure engineering could not. The question “who cannot use this?” is, at bottom, an HCI question.

Dead End: The Interface That Required an Expert

The command-line interface was not simply replaced by the GUI; it was recognized, belatedly, as a usability failure for the general population.

The Expert Assumption

Early computing interfaces were designed on an implicit assumption that users were experts: trained technicians who had read the manual, memorized the command syntax, and were willing to invest hours in learning before gaining any functionality. This assumption was appropriate for the 1960s, when computers were expensive, rare, and operated by professional staff. It became untenable when personal computers reached millions of households whose occupants had no interest in becoming experts.

The UNIX command line, in particular, optimized for experts: short commands (ls, grep, awk) minimized typing for users who already knew what they meant; cryptic error messages encoded internal states rather than user-comprehensible descriptions of what had gone wrong; and the learning curve was steep, steep, and then relatively flat. Expert users could be enormously productive; novice users could be paralyzed. HCI recognized this not as an acceptable tradeoff but as a design failure: an interface that worked only after an investment of learning that most people would not make was, for most people, not an interface at all.

The GUI did not eliminate expertise requirements (expert GUI users are still vastly more productive than novices) but it reduced the investment required for basic functionality. The iPhone reduced it further. The trajectory is not toward zero expertise, but toward interfaces where basic capability is available to anyone with working fingers and functional vision, and where deeper capability is accessible to those willing to learn without requiring an initiation rite.