Turner Whitted and Ray Tracing

Abstract

J. Turner Whitted, an electrical engineer with a background in sonar signal processing, introduced recursive ray tracing to computer graphics at Bell Labs in 1979. Earlier programs traced a ray from the eye to the first surface it hit. Whitted let that ray spawn new ones, a reflected ray, a refracted ray and rays towards each light, and followed each of them in turn, so that a single algorithm produced mirror reflections, transparent glass and sharp shadows. He presented the method at SIGGRAPH in 1979 and published it in Communications of the ACM in June 1980 as “An Improved Illumination Model for Shaded Display.” Its best-known image, two spheres, one of them glass, above a checkerboard, took 74 minutes to compute on a VAX-11/780. Recursive ray tracing, often called Whitted-style ray tracing, became the basis of most software-rendered frames for three decades and, from 2018, of the ray-tracing hardware in consumer graphics cards. Whitted later co-founded the graphics-software company Numerical Design, co-invented the signal processing behind Microsoft’s ClearType, and joined NVIDIA Research in 2014.

The principle of recursive ray tracing: a ray from the eye hits a surface and spawns reflected, refracted and shadow rays, each traced in turn. Image: Wojciech Muła, public domain, via Wikimedia Commons.

Sonar

John Turner Whitted was born in Durham, North Carolina, and grew up in Winston-Salem. He studied electrical engineering at Duke University, where he took his bachelor’s and master’s degrees, and worked on digital test equipment, antenna measurement systems and components of a sonar signal processor. His training, the SIGGRAPH history notes, was in antennas, the stability of nonlinear systems and ocean acoustics. In graduate school at North Carolina State University he moved to image processing and then to computer graphics, which he called “a natural transition,” because much of the mathematics was the same. He received his PhD in 1978.

Signal processing shaped his view of graphics. A computer image is a grid of samples of a continuous scene, and when the scene contains detail finer than the grid, the samples produce aliasing: jagged edges, crawling patterns and flickering in animation. Whitted’s interest in anti-aliasing came from this background, and it was one of the problems his ray tracer was built to address.

Rays

The idea of finding what is visible along lines of sight was old. Albrecht Dürer described perspective devices in the 16th century that amounted to tracing rays from the eye through a picture plane. In 1968 Arthur Appel of IBM published the first computer algorithm that traced a ray from the eye through each pixel to find the nearest surface and then cast a ray towards the light to see whether the point lay in shadow, a method now called ray casting. In 1971 Robert Goldstein and Roger Nagel at the company MAGI used ray casting to render shaded solids and produced a 30-second film on a CDC 6600. Ray casting computed what an observer saw directly but not what appeared in mirrors or through glass. For reflections, graphics programs of the 1970s used environment maps, pictures of the surroundings pasted onto shiny surfaces, which did not show the objects reflecting each other.

Whitted joined Bell Labs after his doctorate. There, while trying to render spheres that reflected one another, he arrived at a recursive procedure of the kind he knew from sonar work. When a ray from the eye strikes a surface, the program computes the local illumination and then generates up to three kinds of new rays: a reflection ray in the mirror direction, a refraction ray through a transparent surface bent according to Snell’s law, and shadow rays towards each light source to test whether the point is lit. Each reflected or refracted ray is traced in exactly the same way, spawning further rays when it hits something, so that the colour of a pixel is computed from a tree of rays. The recursion stops at a fixed depth or when a ray’s contribution becomes too small to matter.

The method handled in one framework effects that had previously needed separate tricks or were impossible: objects mirrored in other objects, refraction through glass, and shadows cast by every object onto every other. The tree of rays also helped with anti-aliasing. Where neighbouring rays disagreed, at an edge or a highlight, the program could send additional rays through that pixel and average the results, adapting the sampling to the image.

The Image

Whitted’s first ray tracer ran on a DEC PDP-11/45 under Unix with a frame buffer of 9 bits per pixel, and his early pictures were monochrome. The program was written in C. His best-known image, two spheres, one mirrored and one of glass, floating above a checkerboard, was computed on a VAX-11/780 at 640 by 480 pixels and took 74 minutes, about a seventieth of a second per pixel on a machine of roughly one million instructions per second. About a quarter of that time, he found later, went into converting floating-point numbers between single and double precision; declaring every variable as double would have made it faster. In 2007, asked for a copy of the image for a history of ACM SIGGRAPH, he changed a couple of lines of the original C program, recompiled it on his desktop PC, and it finished in under six seconds; he first assumed it had crashed. “The code executed 740 times faster than it had in 1979.”

He showed the work at SIGGRAPH in 1979 and published it in the June 1980 issue of Communications of the ACM (volume 23, number 6) under the title “An Improved Illumination Model for Shaded Display.” Testing every ray against every object was the expensive part of the method, and with his Bell Labs colleague Steve Rubin he addressed that problem in a 1980 SIGGRAPH paper, “A 3-Dimensional Representation for Fast Rendering of Complex Scenes,” which organized a scene into a hierarchy of bounding volumes so that a ray could skip whole groups of objects it could not hit. Acceleration structures of this kind remain essential to every ray tracer.

A modern recursive ray tracing of a reflective sphere, showing the reflections and shadows the method produces in one framework. Image: Tim Babb, CC BY-SA 4.0, via Wikimedia Commons.

Whitted also made a one-minute ray-traced animation, The Compleat Angler, dated 1979 in most accounts (1980 in the SIGGRAPH archive). SIGGRAPH’s history credits his papers and the film with inspiring thousands of people to write their own ray-tracing programs.

What Came After

Ray tracing in Whitted’s form produced perfect mirrors, perfectly clear glass and hard-edged shadows, a look that became synonymous with early computer graphics. Robert Cook, Thomas Porter and Loren Carpenter at Lucasfilm extended it in 1984 to distributed ray tracing, which sends many rays per pixel with random variations to produce soft shadows, blurry reflections, depth of field and motion blur. James Kajiya’s rendering equation of 1986 and the path tracing that follows from it generalized the method to all light transport, the basis of physically based rendering (see Pixar and Computer Animation and The CGI Revolution). Feature films moved to path tracing in the 2000s and 2010s, with Monster House (2006) and Monsters University (2013) among the first.

Real-time ray tracing took four decades. A University of Utah team demonstrated interactive ray tracing at about 15 frames per second in 1999, and Intel showed a ray-traced version of the game Quake Wars in 2008. In 2018 NVIDIA introduced graphics processors with dedicated ray-tracing hardware, and Microsoft added ray tracing to DirectX. Such hardware speeds up exactly the steps that Whitted’s method and Rubin and Whitted’s hierarchy introduced: traversing a hierarchy of bounding volumes and spawning secondary rays at each hit (see The GPU Revolution and Jensen Huang and Nvidia). Whitted said he was “still awed by the fact that real-time ray tracing can be done at all.”

Career

In December 1983 Whitted left Bell Labs and co-founded Numerical Design Limited (NDL) in Chapel Hill, North Carolina, with Robert Whitton, to build 3D graphics software toolkits. He was its president and technical director until 1996 and a director until it merged with Emergent Game Technologies in 2005. He also held an adjunct appointment at the University of North Carolina at Chapel Hill.

In 1997 he joined Microsoft Research, where he stayed about fifteen years, founded the hardware devices group, managed the graphics group and several human-computer interaction groups, and co-invented the signal-processing algorithm used in ClearType, Microsoft’s sub-pixel font rendering for LCD screens, a return to the anti-aliasing problems of his early career. He spent about a decade on efforts to bring graphics processors into displays, and in 2014 joined NVIDIA Research.

He received the SIGGRAPH Computer Graphics Achievement Award in 1986 and SIGGRAPH’s Steven Anson Coons Award for lifetime achievement in 2013. He is an ACM Fellow and a member of the National Academy of Engineering, chaired the SIGGRAPH papers committee in 1997, and served on the editorial boards of IEEE Computer Graphics and Applications and ACM Transactions on Graphics. In 2020 he and colleagues published “Origins of Global Illumination” in IEEE Computer Graphics and Applications. Asked in 2018 whether rendering research was finished, he doubted it, and wondered whether a simplifying principle remained to be found.