Skip to content

Benchmarks and Benchmark Gaming

Abstract

A benchmark is a program run to produce a number that can go in an advertisement. The history of computer benchmarks is a sequence of the same three steps: someone writes a small program to compare machines fairly, vendors learn to make that program run fast without making anything else run fast, and a committee writes a larger program with rules. Whetstone (1972), Dhrystone (1984), LINPACK (1979), SPEC (1988), and the TPC (1988) each began as the cure for the previous abuse. The article follows them to the graphics drivers of 2003 and the phones of 2013 that recognized the test and behaved differently while it ran.

Synthetic Programs

Computers were long compared by MIPS, millions of instructions per second, a unit that says little because instructions differ between machines. The first widely used alternative came from Britain. Brian Wichmann at the National Physical Laboratory had collected statistics on which operations real scientific programs executed, using an ALGOL compiler on the laboratory’s English Electric KDF9. The compiler had been written at English Electric’s site in Whetstone, Leicestershire. From the statistics Wichmann wrote 42 ALGOL statements with the same mix, and Harold Curnow of the government’s Technical Support Unit turned them into a usable program in 1972. They published it as “A Synthetic Benchmark” in The Computer Journal in 1976. Whetstone was mostly floating-point arithmetic and mathematical functions, and its results were quoted in thousands of Whetstone instructions per second.

It was also the first benchmark to be defeated by a compiler. When the FORTRAN version was run through IBM’s optimizing compiler, calculations whose results were never printed were removed as dead code, and the program finished early. Curnow rewrote it so that the work could not be skipped.

Dhrystone was the integer counterpart, and its name is a pun on the first. Reinhold Weicker published it in Communications of the ACM in October 1984, in Ada; a C translation by Rick Richardson for Unix made it popular. It contains no floating-point operations. Its result, divided by the 1,757 Dhrystones per second of a DEC VAX-11/780, gave “Dhrystone MIPS”, the figure that appeared in microprocessor advertising for the next decade.

Dhrystone was small enough to fit in a processor’s cache, so it said nothing about memory, and it spent an unrealistic share of its time copying strings. Compiler writers noticed that the strings had fixed lengths and replaced the copy loops with a few word moves, which could overstate performance by more than 30 percent. Version 2.0, issued by Weicker and Richardson in March 1988, was changed to defeat such techniques and was only partly successful.

One Equation, One List

LINPACK was a FORTRAN library for linear algebra written by Jack Dongarra, Jim Bunch, Cleve Moler, and Gilbert Stewart (see Scientific and Numerical Computing). The 1979 users’ guide had an appendix listing how long various computers took to solve a system of 100 linear equations with it, so that users could estimate their own run times. Dongarra kept the list going, and it became a benchmark. In 1991 the rules were changed to allow a problem of any size, so that large parallel machines could be measured.

Hans Meuer at the University of Mannheim had been compiling statistics on supercomputer installations. In 1993 Dongarra joined him with the LINPACK numbers, and since June 1993 the TOP500 list has ranked the 500 fastest reported machines twice a year (see The Supercomputer Era).

One dense system of equations is a narrow test, and Dongarra has said so repeatedly: it rewards processor count and peak arithmetic speed and says little about memory bandwidth or the network between nodes. Machines have been built and tuned with the list position in mind. In 2013 he and Michael Heroux of Sandia proposed a second benchmark, HPCG, based on sparse problems closer to real simulations. The TOP500 still ranks by LINPACK.

Committees

By the late 1980s every workstation maker quoted MIPS and Dhrystone figures that no customer could reproduce. In 1988 Apollo, Hewlett-Packard, MIPS Computer Systems, and Sun founded the Standard Performance Evaluation Corporation to agree on something better. Its first suite, announced in 1989 when the membership had grown to ten companies including DEC and IBM, consisted of real programs, among them a compiler and scientific codes. Results were reported relative to the VAX-11/780 and combined by geometric mean. Later suites followed in 1992, 1995, 1999, 2006, 2017, and 2026.

SPEC’s rules recognize the compiler problem instead of pretending it away. A vendor reports a “base” figure, with the same limited set of compiler flags for every program, and may report a “peak” figure with flags tuned for each program separately.

The database vendors had their own war. A test called TP1, of IBM origin, was run by each vendor on its own terms with nobody checking. On 1 April 1985 Datamation published “A Measure of Transaction Processing Power”, which defined a bank-teller workload called DebitCredit and required the price of the system to be reported with the speed. Its byline was “Anon et al.”; the author was Jim Gray with 24 colleagues from competing firms. The same paper defined a sorting test (see The History of Sorting).

DebitCredit had no referee either. On 10 August 1988 the consultant Omri Serlin persuaded eight companies to found the Transaction Processing Performance Council. TPC-A, based on DebitCredit, followed in November 1989 and TPC-C, an order-entry workload, in July 1992. In April 1993 the Standish Group charged that Oracle had added a “discrete transactions” option to its database to raise its TPC-A numbers and for little else. Oracle replied that it had followed the written specification. The council never ruled on the case, but in September 1993 it adopted Clause 0.2, which forbids products “whose primary purpose is performance optimization of TPC benchmark results”, and it began requiring every result to be approved by a certified auditor before publication.

Recognizing the Test

Tuning a product for a benchmark is one thing. Detecting the benchmark and switching behavior is another, and it has been caught several times.

3DMark03, from the Finnish company Futuremark, was the standard test for graphics cards in 2003. On 23 May 2003 Futuremark published an audit of Nvidia’s Detonator FX drivers. The drivers recognized 3DMark03 and, in eight places, replaced the rendering work it asked for with cheaper hand-written substitutes that looked right only from the benchmark’s fixed camera path. Futuremark released a build that defeated the detection, and the score of a GeForce FX 5900 Ultra fell from 5,806 to 4,679: the detection had raised it by 24.1 percent. A Radeon 9800 Pro from ATI lost 1.9 percent, and ATI admitted to one optimization of its own. In early June, after talks between the companies and their lawyers, Futuremark issued a joint statement with Nvidia saying that the driver design was “an application specific optimisation and not a cheat”.

In 2013 it was phones. Samsung’s Galaxy S4 was found to run its graphics processor at a higher clock rate when certain benchmark apps were started. On 1 October 2013 Ars Technica reported that the Galaxy Note 3 contained a list of benchmark names; when one of them ran, the processor cores were kept at full speed. The same Geekbench test scored 20 percent lower when the phone did not recognize it. Samsung said the phone raised its frequencies for any demanding application. Buyers of the S4 sued, and in 2019 Samsung settled the class action in a deal valued at 13.4 million dollars, of which 2.8 million went into a fund paying claimants about ten dollars each.

Why It Keeps Happening

A benchmark is useful to a buyer only while vendors are not optimizing for it, and it is valuable to a vendor only once buyers look at it. Every benchmark in this article was written by people who knew that. Curnow rewrote Whetstone, Weicker revised Dhrystone, SPEC replaces its suite every few years, and the TPC hired auditors. The defenses that have held up are procedural: real applications in place of synthetic loops, full disclosure of configuration and price, independent audit, and retirement of a test once it has been learned.

📚 Sources