New Computer Evaluation Metrics for a Changing World

Peak performance is the status quo, especially for emerging AI accelerators, as it is easy to calculate, and it showcases maximum speed. Alas, it does not predict actual performance.⁴ The flaw of using purchase price is that it focuses on today’s chip cost rather than lifetime system cost.

Benchmarks such as MLPerf⁸ were invented to improve prediction of real performance. The TPC-C benchmark added maintenance cost to purchase price,⁴ leading eventually to Total Cost of Ownership (TCO):¹

where OpEx (Operational Expenditure) is the cost paid during the lifetime of the chip and infrastructure. It covers electricity costs consumed—including power distribution and cooling—and the cost of datacenter space over the server’s lifetime. TPC-C sets N to three years. With the slowing of Moore’s Law, new server CPUs deliver in excess of 100 cores rather than much faster ones, and performance/cost these days hardly improves. Most software is not designed for numerous cores, so in practice new servers are barely faster than old servers. When combined with small performance/cost gains, servers are replaced less frequently, stretching N to perhaps six to eight years today from three years a decade ago. In some systems today, OpEx is half of the TCO.

A problem with using benchmarks for performance is that they do not age well. As benchmark results affect sales, they immediately become the target of engineering efforts that help benchmarks but not necessarily real programs, so they lose their predictive value if refreshed infrequently. They also often target chip performance rather than systems performance. Popular benchmarks such as Coremark, Dhrystone, Linpack, and SPEC2017 are all cautionary examples.

Running the actual workload, such as production AI training, is obviously more accurate than chip benchmarks. However, it is also important to capture if computers are underutilized or if computation is wasted. Goodput is a networking term that only counts the information bits actually delivered, subtracting the protocol overhead and retransmissions due to failures. We borrow that term here to adjust workload performance by subtracting effort wasted on underutilization or unreliability problems.

As an example of underutilization, AI training and many other applications are bulk synchronous parallel,³ where all the computers in a system operate concurrently for one step and then exchange messages. They all wait until all messages are received, so communication speed is important. As the time per step is set by the slowest computer, load balance is critical to ensure that all computers do useful work. Stragglers can significantly degrade overall performance of large-scale synchronous jobs, such as AI training.²

A final goodput adjustment is to remove wasted work from unreliability. The overhead of software error detection, checkpointing, and error recovery can be substantial. Worryingly, hardware errors are increasing—with many going undetected—as we follow Moore’s Law to tinier transistors.^5,9

The TCO formula implicitly assumes that sufficient datacenter (DC) capacity is available to house new servers, because the TCO formula includes DC provisioning cost. However, building new DCs is not always possible or affordable, especially at large scale, as some local electric utilities have practical limits to the maximum power available. Capital expenditure for DCs is also limited, in part because it needs to be spent up front. The combination of zoning, environmental regulations, competition for electricity from other customers, the desire to reduce dirty energy sources, and limited capital puts significant pressure on building new DCs. Thus, an important new metric to join goodput/TCO is the performance that fits within a current DC’s power envelope.

Not all DC power gets to the servers inside. Power Usage Effectiveness (PUE) is the industry-standard metric of DC efficiency, defined as the ratio between total energy usage (including all overheads, like cooling and power distribution) divided by the energy directly consumed by the DC’s computing equipment. If 1.5W of power must be delivered to the DC to, in the end, deliver 1W of power to a server after accounting for distribution overheads and cooling, then the PUE would be 1.5. This metric rewards DCs that reduce PUE, as they can hold more servers. In 2007, it was 2.5 (150% overhead), but by 2022, cloud providers cut PUEs to 1.1 (10% overhead).⁷ Reducing energy overhead 15x in 15 years illustrates the impact of new metrics.

A simple way to calculate the maximum number of servers per DC is to reduce the electrical power available by the PUE and then divide that result by the maximum electricity consumption per server. The last term is limited by the Thermal Design Power (TDP), the maximum heat a computer can cool. In practice, not all servers in a DC operate close to TDP simultaneously. An optimization is to instead deploy more servers assuming an over-provisioned TDP capacity, called the Oversubscription Rate (OSR).¹ We only need a backoff method for the rare times when the actual power for many servers in a DC is too high, which is much easier for AI workloads, such as training or bulk inference, than for traditional user-facing applications. This insight means that the number of servers per DC can be based more on average power rather than on the worst-case TDP, increasing the importance of reducing average power.

Environmental sustainability is increasingly crucial, which means information technology should minimize its carbon footprint. Measured in carbon dioxide equivalent emissions, CO₂e accounts for other greenhouse gasses, such as methane. The CO₂e unit is metric tons (t), which are 1,000kg. Therefore, an additional important metric to goodput/TCO and goodput/DC power is goodput/CO₂e.

CO₂e can be divided into the operational portion—from running computer equipment—and the embodied portion, from manufacturing it and building DCs. Embodied CO₂e also includes the sourcing of raw materials, upstream energy use by suppliers to the manufacturers, and transportation.

Environmental organizations define rules to ensure that operational and embodied emissions are accounted for and allocated correctly at the proper stages of a value chain. Multiple standards specify boundaries and ensure all emissions are accounted for appropriately. Corporate accounting commonly uses Greenhouse Gas Protocol (GHGP) while product-level carbon footprinting commonly uses ISO standards 14040 and 14044. Both GHGP and these ISO standards encompass all operational and embodied emissions (further refined as scope 1, 2, and 3).

Operational CO₂e use is well understood, and it depends heavily on where the energy is consumed.¹⁰ If the local grid relies primarily on renewable energy sources instead of fossil fuels, the footprint can drop 10x. The unit is grams of CO₂e per kilowatt-hour (kWh). The worldwide average today is 475, but sites can drop below 60 by using solar or wind power and rise above 700 by burning coal.

Unfortunately, the embodied CO₂e of CPU servers and AI accelerators is rarely published. The range of embodied CO2e per server from the limited publications is 1t to 4t.¹¹ Given the high variance, more investigation is necessary. The embodied footprint of DC buildings is also less documented but likely much less than the servers inside given the 20-year amortization of DCs.⁷

Embodied emissions, similar to operational emissions, also heavily depend on where the chips are manufactured, as half is from electricity use.¹² The top countries that manufacture chips are Taiwan, South Korea, and Japan, where the grid carbon intensity is still high at 542, 457, and 594 grams/kWh, respectively. Given these drawbacks to chip manufacturing and the grid decarbonization plans of many countries that deploy most chips, embodied CO₂e is fundamental to the carbon footprint of information technology and needs to be better quantified and tracked.¹¹

To illustrate the metrics and their importance, the last column of the Table compares a hypothetical deployment of two recent AI accelerators: Google’s TPU v3 and TPU v4. Public data⁶ records:

A few assumptions supply the missing parameters:

Performance/cost in the Table swings from peak/purchase price metric value of 1.9x of TPU v4 over TPU v3 to benchmark/TCO metric value of 3.2x.

The other metrics offer much larger gains. By taking advantage of oversubscription for TPU v4 as opposed to a standard allocation for TPU v3, goodput/DC power metric becomes 6x for TPU v4. Even after accounting for the larger energy use and embodied carbon footprint of deploying more servers that oversubscription enables, building a TPU v4 DC near green energy instead of an average location raises the goodput/CO2e metric value to 26x.

It’s hard to improve what you do not measure. The proposed metrics upgrade the conventional performance/cost equation in both the numerator and the denominator. The former improved from peak performance to benchmark performance to goodput, while the latter advanced from purchase price to TCO, DC power capacity, and carbon emissions. To find the best solution, infrastructure architects should co-optimize three metrics: goodput/TCO, goodput/data center power, and goodput/CO₂e.

Given where we are with Moore’s Law, Dennard scaling, and environmental sustainability, our information technology community should: