A Chip That Wins by Refusing to Be General

An AI accelerator is a processor built to perform the matrix and vector arithmetic of machine learning, and it works by dedicating most of its silicon to thousands of simple multiply and accumulate units and to the memory paths that feed them, rather than to the control machinery a general processor carries. A matrix is a rectangular array of numbers arranged in rows and columns, and a vector is a single row or column of numbers. Machine learning models transform vectors through layer after layer of matrix operations, and the accelerator exists to carry out that transformation at high throughput, meaning a large volume of completed operations per second, rather than to handle the varied tasks a laptop or server processor handles minute by minute.

The central claim defended across the sections that follow is straightforward to state and demanding to build. An accelerator gains its advantage by surrendering generality. It spends transistors, the tiny switches etched into silicon, and electrical energy on repeating a single operation across wide arrays, and on moving the numbers for that operation into place without interruption. Because that bargain shapes every later choice, the binding limit on performance is seldom the arithmetic itself. The binding limit is the movement of data between memory and the arithmetic units, and a design that ignores that limit wastes the silicon it saved.

Three questions organise the argument. The first asks what the arithmetic actually is, and why a handful of repeated operations account for nearly all of the computation in a neural network, which is a layered mathematical model whose adjustable numbers are called weights. The second asks what feeding that arithmetic costs, in energy, in time, and in memory bandwidth, where bandwidth means how much data moves per second. The third asks where design choices diverge for training, the process of adjusting weights by measuring errors against known examples, and inference, the process of applying a trained model to fresh inputs to produce a prediction. Each question leads to a different part of the machine, and each part answers to the same memory constraint.

John L. Hennessy and David A. Patterson made the broader case for this kind of specialization in Communications of the ACM in 2019. They argued that domain specific architectures, processors shaped for one class of work, offer a path forward as general purpose scaling slows. An accelerator is that argument rendered in silicon. It removes circuits that a general processor needs for unpredictable programs, enlarges the arithmetic arrays, widens the paths from memory, and accepts that whole categories of software will run poorly or not at all on the resulting chip. The gain and the loss come from the same decision, and neither can be understood apart from the other.

That trade also explains why accelerator designs differ so sharply from one another despite sharing a workload. One design may hold weights stationary inside the array while inputs stream past. Another may favor flexible scheduling over a fixed rhythm. A third may cut number precision, the number of bits used to represent each value, to fit more arithmetic into the same area. Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang and Joel S. Emer surveyed these choices in Efficient Processing of Deep Neural Networks, published in Proceedings of the IEEE in 2017, and warned that flexibility and energy pull in opposite directions. Every section that follows returns to that tension, because the machine that wins on one workload can lose its advantage the moment the workload changes shape.

The Workload Is Mostly Multiplication and Addition

A neural network layer reduces, at the level of computation, to large matrix multiplications and vector operations followed by a simple nonlinear function that reshapes each result. A weight matrix multiplied by an input vector produces a set of summed products, where each output value is the sum of many input values each multiplied by a corresponding weight. A convolutional layer, used heavily in image processing, applies the same pattern with a small weight set slid across an image. A transformer layer, used in language models, multiplies sequences of vectors by several weight matrices in succession. The surface forms differ, and the underlying arithmetic stays remarkably uniform across all three shapes.

The atomic operation inside that arithmetic is the multiply accumulate. One multiply accumulate takes two numbers, multiplies them, and adds the product to a running total held in an accumulator register. Summed products are built entirely from this step, repeated billions of times. By convention, one multiply accumulate counts as two floating point operations, because it performs one multiplication and one addition. That convention matters whenever performance figures are compared, since a chip credited with a given number of operations per second is counting each fused step twice. Honest comparison begins with knowing what the unit of counting actually contains.

The scale of the repetition follows from the shape of matrix multiplication itself. Multiplying two square matrices of side n requires on the order of n cubed multiply accumulates, while the two matrices together hold only on the order of n squared data values. Doubling the side of the matrices multiplies the arithmetic eightfold but the data only fourfold. Each loaded value therefore participates in more operations as the matrices grow, a property called arithmetic intensity when measured as operations per byte moved. That growth is the seed of every later design decision, because it means large regular blocks of arithmetic reward reuse, and reuse is what specialized hardware is built to exploit.

Sze, Chen, Yang and Emer documented this dominance of the multiply accumulate in their 2017 survey in Proceedings of the IEEE, tracing how deep network processing spends the overwhelming share of its operations and its energy on this single repeated step and on the data movement that supports it. Their account also clarifies what the nonlinear function contributes. After each summed product passes through a function such as a rectified linear unit, which passes positive values unchanged and sets negative values to zero, the result becomes an activation, an input for the following layer. The nonlinear step is cheap in operations compared with the multiplication that precedes it, yet it is what allows stacked layers to represent curved relationships rather than one large straight line.

Training adds a second family of matrix work on top of the forward calculation. A gradient is the measure of how much a small change in one weight would change the overall error, and backpropagation computes gradients by propagating the error signal backwards through the network using the chain rule of calculus. Training therefore performs roughly three passes of matrix work, a forward pass, a backward pass for activations, and a backward pass for weights, against one forward pass for inference. The standard approximation used in the survey literature treats those three passes as comparable in size. An accelerator built for training must sustain that triple load and store the intermediate activations the backward passes consume, while an inference device carries a lighter and more predictable burden.

Why a General Purpose Processor Spends Its Budget Elsewhere

A central processor is a machine built for irregular work. Programs for general computing branch unpredictably, depend on values produced moments earlier, touch memory in patterns no schedule can foresee, and interrupt one another constantly. The central processor answers that irregularity with machinery devoted to guessing and recovering. Branch prediction hardware forecasts which way a conditional instruction will go so work can proceed before the answer is known. Out of order execution rearranges instructions so that a delayed memory access does not stall every operation behind it. Speculative paths carry out instructions that may turn out to be unnecessary, discarding the results when a forecast proves wrong. Large caches, fast memory areas close to the core, hold data in case unpredictable accesses return to it.

Following one instruction through such a core shows where the budget goes. The instruction is fetched from memory, decoded into internal operations, checked against hazards that could corrupt ordering, issued to a reservation station, matched with operands that may arrive from a cache or a register file, and only then handed to an arithmetic unit for the fraction of a cycle in which multiplication or addition occurs. Afterward the result is reordered, committed in program sequence, and written back. The arithmetic unit itself is a small part of the machinery the instruction passes through. Most transistors and most energy serve control, the apparatus that decides what happens next and repairs the consequences when a guess fails.

That allocation reflects a deliberate goal. A central processor optimizes latency, the delay before a single result arrives, on one unpredictable instruction stream. A person waiting for a spreadsheet to recalculate or a database to answer cares about the delay on that particular chain of dependent steps. Serial speed, not parallel volume, is the product being sold. The cost is that only a fraction of the chip performs arithmetic on any given cycle, because the rest is occupied predicting, scheduling, caching and checking. For workloads with few branches and vast regular data, that supporting machinery becomes expensive overhead carried into every operation.

Mark Horowitz quantified the energy side of this division in his ISSCC keynote in 2014. His table, given for 45 nanometre technology, placed a 32 bit floating point multiplication at a cost of a few picojoules, while an access to dynamic random access memory, the main memory technology used in computers, cost hundreds of picojoules. The orders of magnitude, rather than any exact figure, carry the lesson. Arithmetic is cheap. Fetching the numbers to operate on is expensive, and a general processor spends heavily on the caches and prediction logic that try to hide that expense for irregular programs.

An accelerator reads the same figures and draws the opposite design conclusion. Where the workload is regular, prediction can be discarded because the sequence of operations is known in advance. Where data arrives in streams, large speculative caches give way to explicitly managed on-chip storage. Where thousands of identical operations wait, control can be shared across a whole array instead of repeated for every element. The silicon and energy released by those deletions become arithmetic units and wider memory paths. The general processor is not badly built. It is built for a different problem, and its budget shows exactly what that problem costs to solve well.

How AI Accelerators Turn Matrices Into Arithmetic

The working path of an accelerator begins outside the chip. A host processor, ordinarily a central processor running the surrounding program, hands the accelerator a description of an operation rather than the operation itself, specifying which matrices participate, where they sit in memory, what shape they have, and where results belong. The accelerator queues that description, fetches the named matrices from main memory into its own on-chip storage, divides the work among its arrays of arithmetic units, accumulates partial sums as products form, and writes finished results back to memory for the next layer or for the host to collect. The host handles irregular duties such as input handling and program flow, while the accelerator handles the dense arithmetic in bulk.

Staging matters because the arithmetic units consume data far faster than main memory can deliver it one element at a time. On-chip storage, built from static random access memory, sits close to the arrays and supplies operands at the rate the arrays demand. A direct memory access engine streams blocks in from main memory while computation proceeds on blocks already resident, overlapping movement with arithmetic so neither waits idle for the other. Partial sums often remain on chip across many steps, since writing a half finished total to main memory and reading it back would spend bandwidth on data that has not yet become a result. The choreography of loads, computation and stores is set in advance by a compiler or scheduler that knows the full shape of the operation.

Parallelism, meaning many operations carried out at the same time on different data, is the property that turns this choreography into speed. A sequential machine applies an operation to one pair of numbers, then the next pair, then the next. An accelerator applies the same operation to thousands of pairs during the same interval, each pair handled by its own arithmetic unit under a shared schedule. Machine learning tolerates this treatment because its core operations have few branches. Every output element of a matrix product follows the identical recipe of multiplied pairs summed in sequence, no element depends on a neighbouring element, and no decision inside the calculation redirects the work. Regularity is the precondition, and matrix arithmetic supplies it in abundance.

Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton demonstrated the practical force of that regularity in ImageNet Classification with Deep Convolutional Neural Networks, published at NeurIPS in 2012. They trained the network known as AlexNet on two Nvidia GTX 580 graphics processors, hardware built to shade pixels in parallel, and showed that graphics hardware could be bent to neural network training with results that reshaped the field. The demonstration mattered for accelerator design because it separated the essential ingredient from the incidental one. The graphics processor was not designed for neural networks. Its wide parallel arrays matched the workload anyway, and purpose built accelerators start from that match and remove everything the workload does not need.

Control inside an accelerator therefore looks different from control inside a central processor. Instead of predicting individual instructions, the accelerator follows a dataflow, the planned pattern by which numbers move between storage and arithmetic units and among the units themselves. A scheduler issues wide instructions that direct whole arrays, sequencers step through tile schedules prepared ahead of time, and counters track addresses in predictable strides. Each arithmetic unit receives the same control signals as its neighbours, so the cost of control is divided across thousands of operations rather than paid once per operation. That division is the mechanism by which deleting generality converts directly into arithmetic density, and it holds only while the workload stays regular enough for a shared schedule to describe.

Tiles and Blocks: Cutting a Large Calculation to Fit the Chip

No chip can hold a whole model layer in its fastest storage. A single weight matrix for a large layer can occupy more capacity than the entire on-chip memory of the accelerator meant to multiply it, and the input activations and partial sums compete for the same limited space. The calculation is therefore cut into tiles, rectangular blocks of the matrices small enough to reside on chip together. Each tile is loaded once, used against many other tiles while it remains resident, and retired only after its contribution is complete. Results accumulate across tile visits, and finished output tiles are written back to main memory to free space for the blocks that follow.

Tiling exists to raise arithmetic intensity, the number of operations performed per byte moved from memory. A value loaded from main memory and used once forces the memory system to deliver one value for every operation or two, a ratio no bandwidth can sustain at accelerator speeds. The same value held on chip and used against a hundred partners spreads its loading cost across a hundred operations. Weights benefit most from this treatment in inference, where one weight tile serves an entire batch of inputs. Input tiles benefit when many weight tiles pass against them. Partial sums benefit whenever they stay in registers or on-chip storage instead of travelling to main memory between additions. The tile sizes a compiler selects balance these three forms of reuse against the fixed capacity of the storage available.

One tile’s journey shows the mechanism concretely. A weight tile arrives from main memory through the direct memory access engine and settles into on-chip storage. An input tile joins it, either freshly loaded or retained from a previous step. The arithmetic array multiplies the resident blocks, element by element across thousands of units, while partial sums gather in accumulators positioned close to the multipliers. A second input tile replaces the first and the same weight tile serves again, multiplying the value of the original load. Only when every partner tile has passed does the weight tile leave, and only when an output tile is complete does it travel back to main memory. Loads, computation and stores overlap throughout, with the schedule arranged so the array rarely waits.

Tri Dao and co-authors provided a worked example of tiling carried beyond matrix products in FlashAttention, published at NeurIPS in 2022. Attention, the mechanism by which a transformer layer weighs the relevance of different positions in a sequence, ordinarily materialises a large intermediate matrix in main memory and reads it back repeatedly. Dao and co-authors tiled the attention calculation so that intermediate results stayed in fast on-chip static memory rather than main memory, recomputing small pieces where recomputation cost less than storage and travel. The result demonstrated that the binding resource was memory movement rather than arithmetic, since the same operations performed in a different order against the same hardware reached the answer with far less traffic.

Tile shape carries its own constraints, and poor choices waste the array they were meant to feed. A tile narrower than the arithmetic array leaves columns of multipliers idle. A tile taller than the on-chip storage forces spills to main memory that erase the reuse the tiling was built to create. Edge tiles, left over when a matrix dimension divides unevenly by the tile size, run at partial efficiency and can dominate the cost of small layers. Compilers therefore search tile dimensions against the measured capacity and array geometry of each target chip, and the same model can receive quite different schedules on different hardware. The calculation is identical in every case. The division of that calculation into movable, reusable blocks decides whether the hardware performs near its arithmetic ceiling or near its memory floor.

Inside the Matrix Unit: Thousands of Small Multipliers

The matrix unit is where the accelerator’s surrendered generality becomes visible as hardware. Rows of multiplier and adder pairs fill the silicon, each pair small and deliberately simple, each performing one multiply accumulate per clock cycle under signals shared with every neighbour. A clock cycle is the basic timing step of the chip, set by the clock rate in cycles per second. Outputs from the pairs travel along reduction paths, structured routes that sum many partial products into fewer totals, until complete output values emerge. There is little in a single unit to admire. The achievement lies in the count, in the feeding arrangement, and in the discipline that keeps thousands of identical steps advancing together without individual supervision.

How does a matrix unit multiply two matrices so quickly?

The unit holds one block of numbers resident in its array while the other block streams past it, and every element performs a multiplication and an addition during the same clock cycle. A whole block product therefore advances in parallel across thousands of elements, rather than proceeding element by element in sequence.

Throughput, the volume of completed operations per second, can be raised by two routes. Raising the clock rate makes each unit work faster, but energy consumption climbs steeply with clock rate and the heat that follows must leave the package through its surfaces. Adding units makes the array wider instead, and each added unit contributes its multiply accumulate at the existing clock rate with modest energy cost per operation. Accelerators choose unit count because their workload offers the parallelism to occupy wide arrays, while a central processor serving serial code gains little from width it cannot fill. Width also scales the memory problem, since every added unit is another mouth the feeding system must supply, which returns the design to the bandwidth question that governs the whole machine.

Sharing control across the array is what makes the width affordable. A central processor core carries its own fetch, decode and scheduling machinery for a handful of arithmetic units. A matrix unit amortises one stream of control signals across an entire grid, so the transistors spent deciding what to do stay nearly constant as the arithmetic count grows. The units themselves shed features as well. Individually they handle a narrow set of number formats, they do not predict branches, and they do not reorder their work. Each deletion frees area, and the freed area becomes more units. Density of arithmetic per square millimetre, rather than sophistication per unit, is the metric the design pursues.

Tensor cores, introduced in Nvidia’s Volta architecture in 2017, stand as a dated instance of the matrix unit idea placed inside a graphics processor rather than a standalone accelerator. A tensor core accepts small matrices as operands and returns a matrix result, performing a block multiply accumulate as a single instruction instead of issuing each scalar operation separately. Placing that function inside a programmable graphics processor showed that the matrix unit could operate as one organ within a flexible machine as well as the whole body of a specialized one. The two placements embody the flexibility and efficiency tension that Sze and co-authors described in their 2017 survey, and both persist because different buyers weight that tension differently.

Reduction paths deserve attention because a summed product is only finished when its parts combine. Multiplication produces partial products in parallel across the array, and those partials must be added in a sequence or tree of additions before an output value exists. Long reduction paths add latency, the delay before a result arrives, and consume wiring and energy of their own. Designers position accumulators close to multipliers, keep partial sums in place where the dataflow allows, and shape the array so reductions travel short distances.

Systolic Arrays Pass Data Like a Heartbeat

A systolic array arranges simple cells in a grid, and each cell receives numbers from its neighbours, multiplies, accumulates, and passes results onward in a steady rhythm set by a shared clock. Data enters at the edges of the grid and moves from cell to cell rather than returning to memory after each use, so a loaded value is used many times during its journey across the array. Each cell holds little: a multiplier, an adder, registers for values in transit, and in many designs a stationary operand that remains in place while other values stream past. The regularity of the motion is the point. Neighbouring cells communicate over short wires, timing stays predictable, and the memory system is touched mainly at the boundaries where fresh data enters and finished results leave.

The name comes from the heartbeat analogy in the original literature, systolic referring to the rhythmic contraction by which the heart pumps blood. The analogy captures the design discipline. Just as a pulse moves blood through chambers in sequence, a clock pulse moves data through cells in sequence, and every cell performs its small task as the pulse passes. No cell decides anything about routing or timing. The pattern of movement, the dataflow, is fixed by the arrangement of the grid and the order in which operands enter, and correctness depends on values arriving at the right cell on the right cycle, a condition the loading schedule guarantees by construction rather than by checking.

Dataflow choices define the variants within the family. In a weight stationary dataflow, each cell holds one weight for the duration of a calculation while inputs stream horizontally and partial sums travel vertically, maximizing reuse of the operand that inference uses most. In an output stationary dataflow, partial sums stay in their cells while weights and inputs both stream, minimizing movement of the accumulating totals. In a row stationary dataflow, introduced by Yu-Hsin Chen and co-authors in Eyeriss: A Spatial Architecture to Optimize the Energy Efficiency of Convolutional Neural Networks at ISCA in 2016, each row of cells holds a sliding window arrangement that balances reuse across weights, inputs and partial sums for convolutional work. Each variant answers the same energy question with a different decision about which operand deserves to stay still.

H. T. Kung and Charles E. Leiserson developed the systolic array concept in the late 1970s, and Kung set out the argument in Why Systolic Architectures?, published in Computer in 1982. Their proposal answered a problem already visible at the time, that processor speed was outrunning the memory connections feeding it, and that algorithms with regular structure could be mapped onto grids of modest processors that passed data among themselves. The idea waited decades for a workload large and regular enough to justify building it at scale. Matrix multiplication for neural networks supplied that workload, and the grid of simple cells reappeared as the arithmetic heart of purpose built accelerators.

Norman P. Jouppi and co-authors analyzed the best known deployment in In-Datacenter Performance Analysis of a Tensor Processing Unit, presented at ISCA in 2017. The first Tensor Processing Unit, deployed in Google data centers from 2015, organized its arithmetic as a large systolic array and delivered inference performance 15 to 30 times that of the contemporary central processor and graphics processor baselines in that study, on that study’s workloads, with far better energy efficiency. The qualification belongs with the figure every time it appears, because different workloads weight the array’s strengths differently. Layers whose shapes match the grid run at high occupancy, while small or irregular operations leave cells idle and erode the advantage. The systolic rhythm rewards regularity with exceptional efficiency and punishes deviation in the same measure, which is the specialization bargain expressed as geometry.

Precision Is a Budget Spent on Purpose

Every number an accelerator handles is stored in a format, a fixed count of bits divided between an exponent, which sets the range of magnitudes the format can represent, and a mantissa, which sets the fineness of the steps between values within that range. The reference format for scientific computing is 32 bit floating point, with 8 exponent bits and 23 mantissa bits. Precision, in this context, means how finely values can be distinguished, and it is purchased with bits. More bits mean larger multipliers, wider registers, wider memory paths and more energy per operation. Fewer bits shrink every one of those costs at once. Number format selection is therefore a budget decision taken before any other optimization begins, because it scales the entire machine.

Bfloat16 illustrates the trade with unusual clarity. The format keeps the 8 exponent bits of 32 bit floating point and cuts the mantissa to 7 bits, giving a 16 bit container with the same range as the reference format and far coarser steps within that range. Range survives intact, so very large and very small values remain representable, while nearby values blur together. Standard 16 bit floating point makes the opposite allocation, keeping 5 exponent bits and 10 mantissa bits, which buys finer steps across a narrower range. Below those formats sit 8 bit formats used widely for inference, and integer arithmetic at the smallest sizes, where values are whole numbers scaled by a factor held separately.

Machine learning tolerates this reduction because its quantities are statistical rather than exact. A weight is one influence among millions on an output, an activation is an intermediate signal rather than a measurement, and errors introduced by coarse representation tend to average out across the vast number of operations in a layer rather than accumulate in one direction. Training protects itself with a standard mixed precision method, in which calculations run in a reduced format while a master copy of the weights is kept in 32 bit floating point, so the small updates that training applies are not lost to rounding when added to weights many orders of magnitude larger. Inference, which applies fixed weights without updating them, tolerates the smallest formats with the least protection, since a prediction needs only enough fidelity to preserve the model’s ranking of outcomes.

The savings compound across the machine. A multiplier for 8 bit values occupies a fraction of the area of a multiplier for 32 bit values, so more units fit on the same silicon and throughput rises without any change in clock rate. Each operation moves fewer bytes, so a fixed memory bandwidth supports more operations per second, and arithmetic intensity measured in operations per byte climbs even though the underlying calculation is unchanged. Energy per operation falls on both counts that Horowitz identified in his 2014 ISSCC keynote, since the multiplication itself costs less and the memory traffic behind it shrinks. Precision reduction is thus the rare optimization that relieves the arithmetic budget and the memory budget simultaneously, and it connects directly to the claim that feeding the array, not computing, binds performance.

Sze, Chen, Yang and Emer treated precision as a first order design variable in their 2017 survey in Proceedings of the IEEE, connecting number format choices to energy consumption across the whole processing chain. Their treatment also marks the limits of the trade. Formats cut too far lose the small gradients that training depends on, quantized models can fail on inputs unlike those used to calibrate their scaling factors, and some operations, such as the accumulation of long summed products, demand wider formats internally even when inputs and outputs are narrow. Accelerators therefore accumulate in higher precision than they multiply, a quiet admission inside the hardware that precision is spent on purpose rather than discarded.

The Roofline: Memory Bandwidth Sets the Ceiling

One table settles the comparison that the surrounding sections build toward, because it settles which resource each accelerator design spends freely and which it starves, so a reader can predict a design’s behavior from its balance rather than its peak rate. The five design approaches in the table differ less in their ambitions than in their scarcities, and naming the scarcity in each case converts marketing language about speed into a testable expectation about where a given layer, model or serving pattern will slow down and why.

Design approach	What it optimizes	What it sacrifices	Binding limit	Workload it fits
General purpose processor	Low latency on irregular code	Parallel arithmetic density	Control and cache overhead	Mixed workloads with unpredictable branches
Graphics processor	Parallel throughput with programming flexibility	Energy per operation against specialized designs	Memory bandwidth at low reuse	Training and varied research workloads
Systolic array accelerator	Data reuse inside a fixed grid	Flexibility for irregular operations	Performance when shapes do not match the grid	Large regular matrix products and inference
Reconfigurable fabric	A data path shaped to one pipeline	Raw clock rate and ease of programming	Development effort for each design	Fixed streaming pipelines with stable shapes
Inference specialized chip	Energy per prediction at small number formats	Training capability and generality	Model changes that outgrow fixed functions	High volume prediction on stable models

The roofline model gives that comparison a quantitative form. Samuel Williams, Andrew Waterman and David Patterson introduced it in Roofline: An Insightful Visual Performance Model for Multicore Architectures, published in Communications of the ACM in 2009. The model plots attainable performance against arithmetic intensity, operations per byte of memory traffic, and caps performance at the lower of two ceilings. The first ceiling is the peak arithmetic rate of the machine, a flat line no workload can exceed. The second ceiling rises diagonally, set by memory bandwidth multiplied by arithmetic intensity, so a workload gains attainable performance as its reuse improves until it reaches the flat ceiling. Whichever ceiling is lower binds, and the chart names the binding resource at a glance.

A worked contrast shows the two regimes plainly. A large matrix multiplication with wide tiles reuses each loaded value across hundreds of operations, sits far along the horizontal axis, and reaches the flat arithmetic ceiling, so adding multipliers genuinely speeds it up. An elementwise operation, which touches each value once or twice, sits near the origin of the chart, and its attainable performance is fixed by the diagonal bandwidth ceiling regardless of how many multipliers the package contains. Doubling memory bandwidth lifts it immediately. The table above predicts the same behavior from the design side, since a machine whose binding limit is bandwidth at low reuse will post strong peak figures and disappointing results on exactly the layers the roofline marks as starved.

Diagnosis therefore precedes enlargement in any serious performance work, and the habit transfers beyond accelerator packages. Performance work on ordinary cloud machines begins in the same place, since Azure VM performance tuning likewise starts by finding which resource actually binds rather than enlarging the wrong one. Inside an accelerator project the equivalent discipline names the ceiling for the workload that matters commercially, measures arithmetic intensity layer by layer, and spends the design budget against the lower ceiling only. Williams, Waterman and Patterson designed the roofline as a communication tool as much as an analysis tool, a single picture that lets architects and programrs argue about the same constraint in the same vocabulary, and its endurance in design reviews reflects how often the binding constraint turns out to be movement rather than arithmetic.

High Bandwidth Memory Stacks the Reservoir Beside the Engine

Every accelerator drinks from a reservoir. Weights, input activations and intermediate results live in main memory, ordinarily dynamic random access memory, and the arithmetic arrays consume that data at rates set by thousands of units working in parallel. Commodity dynamic memory connects to a processor through an interface sized for central processors, which need modest bandwidth delivered with low latency across unpredictable accesses. An accelerator needs the opposite profile, enormous bandwidth delivered in predictable streams. The mismatch is structural. An array capable of trillions of operations per second starves behind an interface designed for a core that issues a handful of requests at a time, and no scheduling cleverness recovers bandwidth the physical connection cannot carry.

High bandwidth memory answers the mismatch by rebuilding the connection itself. Instead of placing memory chips at a distance across a circuit board, the design stacks dynamic memory dies vertically in a tower, connects the dies to one another through vertical wires passing through the silicon, and sets the stack beside the processor on a silicon interposer, a dense wiring layer that carries signals between the two over a very wide interface. Width substitutes for speed. Thousands of narrow connections operating at moderate rates move more data per second than a small number of fast connections, and the short distance between stack and processor keeps the energy per transferred bit low. The technology is standardized by JEDEC, the industry body whose specifications allow multiple manufacturers to supply interchangeable stacks.

The packaging route is what makes the arrangement physically possible, since placing a memory tower within millimetres of a large processor package demands interconnection far denser than a circuit board provides, and accounts of how chiplets work describe the same packaging family that lets stacked memory and processor dies share one substrate as neighbours. The interposer carries the wide interface between them, the stacks and the processor mount on it as separate dies, and the assembly behaves electrically as one unit while remaining manufacturable as several. That division matters for yield as well as bandwidth, because a defect in one memory die or in the processor die does not condemn the other components before assembly joins them.

The trade carries real costs, and designers quote bandwidth per watt rather than bandwidth alone because the costs concentrate in energy and heat. Stacked dies trap heat in their middle layers, where it escapes slowly, and the stack’s capacity per package remains smaller than what board mounted memory can provide at lower cost. Models or datasets larger than the stacks hold must spill to slower memory beyond the package, and performance falls sharply at that boundary. Bandwidth per watt captures the governing compromise, since a data center pays for electricity and cooling on every transferred byte, and an interface that doubles bandwidth while tripling energy per bit loses money at scale even as it wins benchmarks.

Commodity interfaces remain the right choice where their profile fits, which clarifies what the stacked design actually purchases. A central processor serving branch heavy programs values latency and capacity over streaming width, and board mounted dynamic memory delivers both cheaply. An accelerator serving matrix arithmetic values width above all other properties, accepts constrained capacity as the price, and organizes its tiling, precision and dataflow decisions to live within that capacity. High bandwidth memory therefore completes the argument that runs through every preceding section. The arithmetic was never the scarce resource. The reservoir, the width of the channel between reservoir and engine, and the energy charged per unit of flow decide what the engine can sustain, and the stack beside the processor is the industry answer to a constraint that arithmetic alone could not relax.

On-Chip Storage and the Discipline of Reuse

An accelerator holds data in layers that differ sharply in speed, size, and energy cost. Closest to the arithmetic units sit registers, tiny stores holding the values an operation consumes in the immediate cycle. Beyond the registers lies shared on-chip static memory, a larger but still limited store holding tiles of matrices and partial results awaiting completion. Beyond the package lies main memory, far larger and far slower to reach, where complete models and data sets reside. A design that fetches the same value from main memory for every use wastes both budgets, while a design that fetches a value once and reuses it many times on chip protects both.

The middle layer explains much of the difference between an accelerator and a general processor. A general processor relies on an automatic cache, meaning hardware that guesses which values will be needed again and keeps those values close without direction from the program. An accelerator instead treats on-chip static memory as a managed scratchpad, meaning a store whose contents the programr or the compiler selects explicitly. The distinction matters because matrix work follows a regular pattern that can be planned in advance. The compiler knows the shape of each tile, the order of consumption, and the point at which a partial result becomes complete. Guessed reuse suits irregular work.

Explicit control allows the schedule to guarantee residence rather than hope for residence. A tile placed in the scratchpad stays there until the program replaces it, so bandwidth, meaning how much data moves per second, can serve values not yet used rather than values fetched again. The guarantee also simplifies timing. A cache miss arrives as a surprise that stalls arithmetic units, while a scratchpad schedule exposes every transfer in advance and allows transfers to be arranged around computation. The cost is responsibility. Software must decide what earns space in a scarce store, how long each value remains, and which value leaves when space runs short.

Three reuse patterns recur in every effective design, and the scratchpad exists to serve all three. The first applies one weight against many inputs, as when a single row of a weight matrix processes a stream of examples in succession. The second applies one input against many weights, as when a single activation contributes to many output values. The third keeps partial sums on chip until complete, so each contribution is added where the running total already sits rather than shipped to main memory and returned.

Yu-Hsin Chen and co-authors addressed the same problem from the arrangement of processing elements in Eyeriss: A Spatial Architecture to Optimize the Energy Efficiency of Convolutional Neural Networks, presented at ISCA in 2016. Eyeriss introduced the row stationary dataflow, a dataflow being the pattern that determines where data moves and where data rests during a calculation. Under row stationary mapping, each processing element receives a row of each operand and holds those rows in place while computation proceeds. A row of weights meets successive portions of input without leaving the element, and a row of input meets successive weights under the same discipline. Partial sums accumulate near the elements that produce them, turning reuse into a property of the mapping itself.

The discipline generalises beyond a single architecture. Registers reward reuse within an operation, the scratchpad rewards reuse within a tile, and careful tiling rewards reuse across a complete layer. Designers measure success by how many multiply and accumulate operations each byte fetched from main memory supports, a ratio that rises when tiles fit the scratchpad and falls when tiles spill. Spilling forces values back to main memory and returns them later at full cost, erasing the benefit on-chip storage was built to provide.

The Compiler Decides What the Hardware Actually Sees

Hardware supplies arithmetic units, storage, and paths between them, but software decides which work reaches those units in which order. A model arrives as a description of operations on tensors, arrays of numbers with named dimensions, written without reference to tile sizes, memory addresses, or transfer schedules. The compiler or the runtime library translates that description into a concrete execution plan. Identical silicon can therefore produce very different results under different plans, because the units spend different shares of time waiting, repeating transfers, or handling data arranged in an order the units cannot consume directly.

Tile size selection is the first major choice. A matrix multiplication divided into tiles must fit its working set inside on-chip storage while exposing enough parallel work to occupy the available units. Tiles that are too small cause weights and inputs to be fetched again for each neighbouring tile, multiplying traffic. Tiles that are too large exceed the scratchpad and force spills to main memory, with the same penalty arriving by a different route. The compiler weighs tile dimensions against matrix shape, because a tile that suits a square matrix may suit a tall and narrow matrix poorly.

Layout transformation addresses the order in which values sit in memory. Arithmetic units consume matrices in a fixed pattern, often row by row or block by block, and a matrix stored in a different order must be gathered element by element at high cost. The compiler can rearrange a matrix once, before execution, so sequential addresses deliver values in the exact order of consumption. Rearrangement costs a pass over the data, yet the cost is repaid when the rearranged matrix serves many operations. Weight matrices benefit most often, because weights change rarely during inference and can be prepared in the preferred layout.

Operation fusion removes intermediate traffic by combining operations that a literal translation would execute separately. A matrix multiplication followed by an addition and a nonlinear function can run as separate passes, with the full intermediate result written to memory after each pass and read back for the next pass. Fusion runs the steps as one operation, so each intermediate value passes directly from the arithmetic unit to the next step while still on chip. Fusion also raises storage pressure, because a fused operation holds values from several steps at once, so the compiler fuses where the traffic saving outweighs the storage cost.

Scheduling completes the translation by overlapping movement with arithmetic. Transfers from main memory take time that arithmetic units would otherwise waste, so the compiler issues a transfer for a future tile while the units process a present tile. Double buffering, meaning the use of two scratchpad regions in alternation, supports the overlap by allowing one region to fill while the other empties into computation. Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer examined that combined hardware and mapping problem in Efficient Processing of Deep Neural Networks: A Tutorial and Survey, published in Proceedings of the IEEE in 2017.

Delivered performance therefore reflects a negotiation between description and machine. A program that ignores tile shape, layout, fusion, and scheduling can leave most capacity unused even though every arithmetic unit meets specification. A program that handles all four well can raise delivered throughput, meaning the amount of completed work per unit of time, severalfold on the same device by reducing waits and repeated movement rather than by accelerating any single operation. The mechanism explains why libraries and compilers carry such weight in accelerator design. Silicon sets the ceiling, and the translation layer decides how close ordinary work comes to that ceiling on each layer shape and batch shape a model presents.

Training Is a Different Job From Running a Model

Training and inference use the same matrices and the same arithmetic units, yet the two workloads place different demands on every part of an accelerator. Training, meaning the process that adjusts a model by using labeled examples until predictions improve, changes the weights themselves. Inference, meaning the running of a finished model on an input to produce an output, leaves the weights fixed and only evaluates. Evaluation requires one pass through the network. Learning requires that pass, a record of everything that happened during that pass, a second calculation that assigns error, and an update applied to every weight. Memory, arithmetic, and time all expand under the second workload.

The forward pass in training resembles inference in structure and differs in obligation. Each layer computes activations from inputs and weights, and each activation must be retained rather than discarded after use. Retention is necessary because the backward pass needs the original activations to compute how each weight contributed to the output. A deep network therefore stores activations for every layer, for every example in the batch, for the full duration between the forward calculation and the corresponding backward calculation. Activation storage often exceeds weight storage, especially with large batches or large inputs such as images with many pixels per dimension. The budgets must accommodate weights, activations, gradients, and updated weights at different points in the cycle, and peak demand decides whether a given model and batch fit on a given device.

Gradient computation adds the second major load. A gradient, meaning a value that states how a small change to a weight would change the error, must be produced for every weight in the model. Producing gradients requires matrix work comparable in scale to the forward pass, performed in reverse order through the layers. A weight update then applies a correction to each weight, and many training methods keep additional values beside each weight, such as a running average of past gradients used to steady the update direction. Each additional stored value multiplies the memory held per weight. The standard approximation used in the survey literature counts roughly three passes of matrix work in training against one forward pass in inference, covering the forward pass, the backward pass for activations, and the backward pass for weights.

Time tolerance separates the workloads as sharply as memory does. Training runs tolerate hours or weeks of computation because the result is a finished model that will serve many later requests. A design for training therefore rewards throughput, meaning completed work per unit of time, above other qualities, accepting high energy use and large hardware in exchange for completing more examples per hour. Inference operates under opposite pressure. Each request arrives with a delay budget, and latency, meaning the delay before a result arrives, decides whether an answer remains useful. A classification or a generated passage that arrives after a long wait fails the purpose of the request even when the answer is correct. Energy per answer also matters directly, because the cost repeats with every request.

Hardware emphasis follows from those pressures. Training systems prize large memory capacity, high arithmetic throughput, and fast links between devices, because a training run must hold activations and gradients while sustaining matrix work across many chips. Inference systems prize fast access to weights, modest memory sized to a single model, and efficiency at small batch sizes, because a request may arrive alone and cannot wait for companions to fill a batch. Some devices serve both workloads, and the balance struck inside such a device reflects a compromise. Capacity added for training activations sits idle during inference, while design choices tuned for a single request limit the batch sizes that training can exploit. The two jobs share arithmetic and diverge in almost everything else.

Gradients and the Arithmetic of Learning by Correction

Backpropagation is the bookkeeping procedure that turns a single measure of error into a correction for every weight in a network. The forward pass produces a prediction and records the activations at each layer along the way. Comparison between the prediction and the correct answer yields an error value at the output. The task of learning is to divide responsibility for that error among millions or billions of weights, including weights in early layers whose influence passed through many later transformations before reaching the output.

The chain rule of calculus supplies the systematic method. A network is a composition of functions, where the output of one layer becomes the input of the next, and the chain rule states how the sensitivity of a final result to an early quantity can be computed by multiplying sensitivities along the path between them. Applied layer by layer in reverse, the rule converts the error at the output into an error signal for the layer before it, then for the layer before that one, until every layer holds a signal matched to its own activations. Each weight then receives a gradient formed from the error signal arriving at its output and the activation that entered through that weight. Each weight moves by a small step in the direction opposite to its gradient, so weights contributing most to the error change most.

Why does training a model need far more arithmetic than running it?

Training runs the model forward, then propagates the error backwards to compute a correction for every weight, then updates the weights, so each example demands roughly three passes of matrix work plus the storage of intermediate results, against a single forward pass for a prediction.

The arithmetic multiplies because each direction of travel carries matrix work at full scale. The forward pass multiplies activations by weights at every layer. The backward pass performs two further families of multiplication at every layer, one that propagates the error signal to the previous layer and one that forms weight gradients from the signal and the stored activations. A layer that cost one large matrix operation in the forward direction therefore costs roughly three large matrix operations across a complete training step. Depth compounds the total, because every layer repeats the pattern, and batch size compounds the total again, because every example in the batch requires a forward record and contributes to the averaged gradient that drives the update.

Stored activations dominate memory for a related reason. Gradient formation at a layer needs the activations that entered that layer during the forward pass, and those activations cannot be reconstructed from the weights alone once later layers have transformed them. The forward pass must therefore write activations aside and hold them until the backward pass reaches the corresponding layer, which for early layers means holding values across almost the entire step. Large inputs produce large activations, wide layers produce wide activations, and large batches multiply every activation by the number of examples carried together. Weights are read repeatedly but stored once.

Gradient checkpointing offers a deliberate exchange for cases where activation storage exceeds available memory. Selected activations are saved as checkpoints and the remaining activations are discarded, then recomputed during the backward pass by running the forward calculation again from the nearest checkpoint. Recomputation costs additional arithmetic, typically a fraction of an extra forward pass depending on checkpoint spacing, and releases the memory that the discarded activations would have held. The trade suits training because arithmetic units often have spare capacity relative to memory capacity, and because a run that fits in memory at higher arithmetic cost completes faster than a run forced into a smaller batch or into transfers to slower storage.

Inference and the Cost of a Single Request

Inference at the scale of a single request reverses the conditions that make large batches efficient. A batch of one presents each weight for a single use. The weight must travel from main memory to the arithmetic units, participate in a small number of multiply and accumulate operations, and then make room for the next weight, with no second example waiting to reuse the same value. Arithmetic intensity, meaning the number of operations performed for each byte fetched, collapses toward its minimum. Under those conditions the ceiling described by the roofline model binds hard. Samuel Williams, Andrew Waterman, and David Patterson presented that model in Roofline: An Insightful Visual Performance Model for Multicore Architectures, in Communications of the ACM in 2009, capping attainable performance at the lower of peak arithmetic and bandwidth multiplied by arithmetic intensity. At low intensity the bandwidth term governs, and added arithmetic units contribute nothing.

Weight streaming dominates the single request regime because model size dwarfs request size. An input may contain hundreds or thousands of values, while the model that processes the input contains millions or billions of weights. Reading the model therefore costs far more movement than reading the request, and the reading repeats for each request served. Caches and scratchpads help only where reuse exists within the request, such as an activation used by many weights or a weight used across many positions in an image. For a fully connected layer evaluated once, reuse within the request is limited, and most weights arrive from main memory exactly once per request.

Design priorities shift accordingly. Inference hardware prizes bandwidth above peak arithmetic, because bandwidth decides how fast weights can stream. Small number formats receive strong emphasis because precision, meaning the number of bits used to store each value and the fineness of the values those bits can represent, directly determines how many bytes must move. A weight stored in 8 bits moves in a quarter of the time required by the same weight stored in 32 bits, and the model streams proportionally faster where reduced precision preserves acceptable output quality. Low energy per answer completes the priority list, since a device serving requests continuously converts energy per answer into operating cost and heat. Peak arithmetic recedes in importance because a single request cannot supply enough parallel work to occupy a large array.

Autoregressive generation displays the pattern in its purest form. Generation of that kind produces one token, meaning one unit of output such as a word fragment, at a time, and each token becomes part of the input for the next step. Every generated token requires the model weights to be read again, because the next step runs the model anew on an extended input. A passage of hundreds of tokens therefore requires hundreds of readings of the weight set, reduced only by whatever portion fits in fast on-chip storage. Generation speed tracks memory bandwidth closely for that reason. Doubling arithmetic capacity without changing bandwidth leaves generation speed nearly unchanged once the array waits on weight delivery, while raising bandwidth raises the rate at which steps can proceed.

The same device can present very different strength at different scales without any inconsistency. A data center evaluation that groups many requests into large batches raises arithmetic intensity, reuses each streamed weight across the batch, and allows peak arithmetic to govern results. The difference lies in workload shape rather than in hardware, and comparisons that ignore batch size compare different ceilings rather than different devices. Careful reading of any inference figure therefore asks how many requests shared each pass of the weights, how large the model was relative to on-chip storage, and which number format carried the weights, because those conditions decide which limit the measurement actually reached.

Batching Trades Waiting Time for Throughput

Batching groups several requests into a single execution so that one pass of the weights serves every request in the group. The mechanism is direct. Weights streamed from memory for a batch of one support one set of operations, while the same weights streamed for a batch of thirty two support thirty two sets of operations at almost the same movement cost. Arithmetic intensity rises with batch size, utilization of the arithmetic array rises with intensity, and the cost of reading the model is divided across every answer produced. Batching therefore converts a memory bound workload into a workload that can approach the arithmetic ceiling, provided the group is large enough and the grouped requests align well enough to share execution without excessive padding or idle lanes.

The gain arrives with a cost charged to each individual request. A request that joins a batch waits for the batch to form, waits while earlier work completes, and receives an answer only when the grouped execution finishes. Latency, meaning the delay before a result arrives, spreads across the group in two senses. The average delay grows because formation time and grouped execution time replace immediate individual execution, and the variation in delay grows because a request arriving just after a batch departs waits longer than a request arriving just before departure. Services with strict delay budgets therefore cap batch size even when larger batches would raise throughput further. Batch size stands as the named control in the trade, a single setting that moves a system between fast answers for few requests and slower answers for many requests served in total.

Queueing intuition explains why the trade is so persistent. A server that waits to fill a vehicle before departure moves more passengers per trip and uses fuel and driver time more efficiently per passenger, while each passenger waits for the vehicle to fill and accepts a departure time set by the group rather than by personal arrival. A server that departs on each arrival pleases each passenger and wastes most vehicle capacity on every trip. Inference serving faces the same arithmetic in a different medium. The vehicle is a pass through the model weights, the passengers are requests, and the capacity wasted by immediate departure takes the form of arithmetic units idling while weights stream past for a single use.

Continuous batching refines the schedule without changing the underlying exchange. In a conventional grouped schedule, a batch forms, executes to completion, and only then admits further requests, so a request that finishes early leaves a lane idle while longer requests in the same group continue. Under continuous batching, requests join and leave the group as sequences finish, so a completed request releases a lane that a waiting request can occupy during the same overall execution period. The refinement raises utilization during generation workloads where requests produce different numbers of tokens and would otherwise hold an entire batch hostage to the longest sequence. Formation delay still applies to arriving requests, and the group still shares each pass of the weights, so the latency cost of grouping remains even as idle lanes shrink.

Operators tune the trade against the character of the workload served. Interactive workloads with human readers waiting on answers favor small batches and accept lower utilization as the price of responsiveness. Offline workloads that process large collections of prompts, images, or documents without a waiting reader favor large batches and accept long individual delays as the price of total volume. Mixed workloads divide capacity between the two patterns or adjust batch size with load, allowing larger groups when queues are long and smaller groups when queues are short.

Sparsity and the Value of Skipping Zeros

A trained network contains many values that are zero or close to zero, and each zero presents an opportunity with conditions attached. A multiplication by zero contributes nothing to a sum, and an addition of zero changes nothing, so a calculation that could skip every zero operand would save arithmetic, movement, and energy in proportion to the share of zeros present. Sparsity, meaning the share of zero values in a weight matrix or activation set, measures the size of that opportunity. Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer treated sparsity as an efficiency lever in Efficient Processing of Deep Neural Networks: A Tutorial and Survey, published in Proceedings of the IEEE in 2017, with the caution that the lever moves delivered performance only when hardware and data representation support skipping in practice.

Zeros arise by two main routes. Pruning, a training technique that removes weak connections, creates zeros by design. A pruning procedure measures the contribution of weights or groups of weights, deletes those judged least valuable, and trains the remaining network further so surviving weights compensate for the loss. Repeated cycles can remove a large share of connections from some layers while output quality stays close to the unpruned result, because many weights carry small values that contribute little individually. Activation functions that output zero for negative inputs convert roughly half of a layer output to zeros under typical input distributions, so activation sparsity appears afresh for every input even in a dense model whose weights contain no zeros at all.

Unstructured sparsity scatters zeros irregularly through a matrix, with no pattern that predicts where the next nonzero value sits. Storage savings arrive readily, because a format that records only nonzero values and positions can be far smaller than a dense matrix. Time savings arrive with difficulty. Hardware must locate each nonzero value, fetch the matching partner value from an irregular position, and route the product into the correct partial sum, and the bookkeeping for scattered values can consume the time that skipping was meant to save. An accelerator without dedicated support for scattered patterns may execute a sparse matrix more slowly than the dense equivalent, because dense execution follows a predictable schedule while sparse execution pays indexing costs on every operation that remains.

Structured sparsity removes whole rows, whole blocks, or fixed patterns within each block, so the remaining work keeps a regular shape. A layer with half of its channels removed is a smaller dense layer, and ordinary dense hardware executes the smaller layer at proportionally higher speed with no indexing burden. A pattern that keeps a fixed number of nonzero values in each small block offers a middle course, regular enough for hardware to skip the absent positions by schedule rather than by search. The cost of structure is freedom. Pruning that must respect block boundaries removes some valuable weights and keeps some weak weights, so a structured result at a given sparsity level generally preserves slightly less quality than an unstructured result pruned without constraints.

Hardware support therefore decides whether skipped zeros become saved time or merely saved storage. Support takes several forms, including storage formats that encode positions compactly, schedulers that balance nonzero work evenly across processing elements, and arithmetic paths that can advance past zero operands without issuing the corresponding fetch. Load balance deserves particular attention, because scattered nonzero values cluster unevenly and a schedule that assigns equal matrix regions to each element leaves elements holding dense regions busy long after elements holding sparse regions finish. Absent such support, sparsity remains a property of the model rather than a property of the execution, and a claim about time saved needs evidence about the execution path rather than about the model alone.

Interconnects Turn Many Chips Into One Machine

A single chip holds a limited amount of memory and delivers a limited amount of arithmetic, and large models exceed both limits. Division of the work across many chips addresses both problems at once, and division creates a fresh problem in return. Chips that share work must exchange values, and the exchange consumes time on links that are slower than the paths inside a chip. Parallelism, meaning the division of a calculation among many units that work at the same time, therefore trades a capacity problem for a communication problem whose size depends on how the division is drawn.

Data parallelism divides the examples and keeps the model whole. Each chip holds a complete copy of the model and trains on a different set of examples, so forward and backward passes proceed independently on each device for the duration of a step. Independence ends at the update. Every copy must apply the same correction, which means the gradients computed on separate chips must be combined before any copy advances. All-reduce, the reconciliation step in data parallelism, performs that combination by summing gradients across all participating chips and distributing the total back to every chip, so each device holds identical updated weights at the start of the next step.

What happens when a model is too large to fit on one chip?

The model is divided across several chips, with different layers or different slices of each layer held on different devices, and the partial results travel between chips during every pass, which makes the speed of the connecting links as important as the speed of the arithmetic.

Model parallelism divides the model itself. Different layers rest on different chips under layer division, so an activation produced on one chip travels to the next chip for the next layer, forward during evaluation and backward during training. Slices of a single layer rest on different chips under tensor division, so a large matrix multiplication splits into partial products that must be combined before the layer output is complete. Both forms place communication inside every pass rather than only at the update step, and both forms require the receiving chip to wait for transmitted values before computation can proceed. The placement of divisions therefore shapes performance directly. A division drawn where activations are small moves fewer bytes than a division drawn where activations are large, and a division that balances arithmetic across chips avoids idle time that no link speed can recover.

Links form a hierarchy with falling bandwidth at each step outward. Connections inside a package move data across millimetres of silicon at the highest rates. Connections inside a server move data across a board or through a dedicated fabric between packages. Connections across a rack, and then across rows of racks, move data through switches and longer cables at lower rates per chip and with higher delay per transfer. System design follows the hierarchy by placing the most frequent communication on the fastest links. Tensor divisions that exchange values many times per step belong inside a package or inside a server where bandwidth is plentiful, while data parallel groups that communicate once per step can span racks with less penalty.

The hierarchy also governs failure and scheduling at scale. A job spread across many chips advances at the pace of the slowest participant in each collective step, so a single delayed chip delays every chip waiting on the combined result. The central lesson repeats the pattern of accelerator design at a larger scale. Arithmetic capacity can be multiplied by adding chips with predictable effect, while delivered performance depends on whether data movement between chips, at every level of the hierarchy, keeps pace with the arithmetic that the added chips provide.

Scaling Out and the Serial Fraction That Refuses to Shrink

Adding chips raises the arithmetic available to a training run, and the rise tempts a simple expectation that twice the hardware finishes the work in half the time. Gene Amdahl addressed that expectation directly in 1967 at the AFIPS Spring Joint Computer Conference, where Amdahl stated the limit imposed by the serial fraction of a workload. Any computation contains a portion that cannot be divided among parallel units, because steps depend on earlier results, because data must be loaded before use, or because participants must coordinate before proceeding. That serial portion proceeds at its own pace regardless of how many units handle the parallel portion. As parallel units multiply, the parallel portion shrinks toward zero duration, while the serial portion remains, and total speedup approaches a ceiling set entirely by the serial share.

The arithmetic of the ceiling deserves attention because the ceiling arrives earlier than intuition suggests. A workload with a serial fraction of five percent can speed up by no more than twenty times, even with unlimited parallel hardware, because the serial five percent alone consumes the time that a twentyfold speedup would allow for the whole job. A workload with a serial fraction of one percent caps at one hundred times. Training workloads accumulate serial and coordination costs from several sources at once. Data must be read from storage and distributed before steps can begin. Gradients must be reconciled across devices before updates apply. Checkpoints must be written without corrupting the run. Control software must track thousands of devices and respond when one device lags or fails.

Strong scaling and weak scaling describe two different questions that added hardware can answer. Strong scaling holds the problem size fixed and asks how much faster the same job completes as devices are added, a direct test of the Amdahl ceiling. Weak scaling grows the problem size in proportion to the number of devices and asks whether completion time stays constant, a test of whether a system can handle larger models or larger batches rather than the same work sooner. A system can perform well under weak scaling while disappointing under strong scaling, because weak scaling gives each added device fresh parallel work to absorb coordination costs, while strong scaling divides a fixed amount of parallel work ever more thinly until coordination dominates each device share.

Diminishing returns appear predictably as a cluster grows for a fixed job. Early additions divide large blocks of parallel work and repay their cost handsomely. Later additions divide already small blocks, and each device receives a thinner slice of arithmetic while still bearing a full share of communication, synchronization, and waiting. Utilization falls across the system, energy per completed example rises, and the marginal device contributes less completed work than the device added before it. Operators respond by matching job size to cluster size, by enlarging batches where optimization allows, and by overlapping communication with arithmetic so reconciliation hides behind useful computation where dependencies permit.

Large systems therefore spend a growing share of capacity on coordination and movement rather than on arithmetic, and the pattern returns attention to data movement as the governing concern of accelerator design at every scale. Inside a chip, scratchpad discipline decides whether arrays stay fed. Between chips, link hierarchy decides whether divided work stays balanced. Across a cluster, the serial fraction decides whether added arithmetic becomes added progress or added waiting. Amdahl stated the underlying principle in 1967 for general computing, and the principle applies with special force to training systems because training multiplies devices, steps, and reconciliation events together. Hardware counts describe potential. The serial fraction, the link speeds, and the movement schedule together describe how much of that potential a real run obtains.

Power, Heat and the Building Around the Chip

Every operation on an accelerator costs energy, and every unit of energy drawn by a board leaves that board as heat. A data center accelerator board drawing hundreds of watts therefore behaves as a heating appliance that also performs arithmetic, and a rack holding dozens of such boards concentrates that heat in a narrow vertical space. Air entering the rack at a controlled temperature leaves far hotter, and the building must move that heated air away, reject the heat outside, and supply cooled air again without interruption. Power and cooling cease to be background services at that density. They become part of the computing system itself, with a capacity limit that can bind before arithmetic capacity does.

Mark Horowitz, in Computing’s Energy Problem (and What We Can Do About It), delivered as an ISSCC keynote in 2014, supplied the comparison that explains why the building matters so much. At the 45 nanometre process scale in his table, a 32 bit floating point multiplication cost a few picojoules, while an access to dynamic random access memory cost hundreds of picojoules. The gap between those figures, roughly two orders of magnitude in his accounting, shows that moving a value costs far more energy than multiplying it once it arrives. A design that fetches operands repeatedly therefore spends its power budget on transport, heats the board through that transport, and gains little arithmetic for the expenditure.

Dynamic voltage and frequency scaling, the adjustment of supply voltage and clock rate in response to workload and temperature, mediates between performance and heat. Raising frequency allows more operations per second, yet power rises steeply because dynamic power depends on voltage squared multiplied by frequency, and higher frequency generally demands higher voltage. Lowering both settings reduces throughput, the amount of completed work per second, while cutting power by a larger proportion. Controllers apply this trade continuously, lifting clocks when thermal headroom exists and reducing them when sensors approach a limit, so the rate printed in a specification describes a ceiling rather than a guaranteed operating point under every load and every room condition.

Thermal throttling is the protective response that follows when cooling cannot remove heat as fast as the silicon generates it. Temperature sensors on the die trigger a reduction in voltage and frequency, performance falls, and the device protects its transistors and solder joints from damage caused by sustained overheating. The mechanism preserves the hardware by sacrificing speed, which means a benchmark lasting seconds and a production run lasting hours can report very different rates on identical equipment. Sustained work settles at the rate the cooling system can support indefinitely, and that settled rate, rather than any brief peak, determines how much computation a facility delivers across a full day of operation.

The rack therefore depends on power delivery and cooling systems sized as carefully as the accelerators themselves, a dependence described in how data centers work in terms of the electrical and thermal infrastructure behind the machines. Uninterruptible supplies bridge brief interruptions, distribution units convert and route power at rack scale, and chilled water or direct liquid loops carry heat where air alone reaches its limit. Each layer adds its own energy cost, measured at facility level by comparing total building consumption against the share consumed by computing equipment itself. An accelerator that halves energy per operation inside the chip but demands disproportionate cooling outside it improves only part of that ledger, and honest accounting keeps both parts in view together.

Benchmarks Without Method Are Marketing

A peak floating point rate describes arithmetic units under perfect feeding, with operands arriving on every cycle, matrix shapes matched to the hardware, and no pause for memory, coordination or input. Delivered performance describes something else entirely. It is a property of a workload, meaning a defined set of calculations on defined data, together with a memory system, a compiler that translates program instructions into hardware operations, and a batch size, meaning the number of inputs processed together before results return. Changing any one of those elements changes the measured result, often by a large factor, while the peak rate stays fixed on the specification sheet. A number reported without its method therefore carries almost no information about the work a machine will complete in service.

A responsible benchmark report states the conditions that produced its figures, in full, beside the figures themselves. It names the workload and the model, gives the size of that model, identifies the number format used for the arithmetic, and reports the batch size at which measurements were taken. It describes the hardware configuration, including memory capacity and the number of devices cooperating on the task, and it names the baseline system against which comparison is made. Precision, the fineness with which numbers are represented, belongs in that list as well, because arithmetic in a reduced format runs faster and moves less data than the same arithmetic carried out at higher precision, making an unqualified speed comparison between formats meaningless.

Why do headline floating point rates overstate real performance?

Headline rates assume every arithmetic unit receives data on every cycle and never waits. Real workloads stall on memory transfers, present matrix shapes that fit the hardware poorly, and lose time to coordination between devices, so delivered performance follows the slowest part of the path rather than the peak rate of its fastest component.

Norman P. Jouppi and co-authors set the standard for this discipline in In-Datacenter Performance Analysis of a Tensor Processing Unit, presented at ISCA in 2017. That study named its workloads, named the central processor and graphics processor baselines used for comparison, described the deployment setting inside Google data centers, and reported energy efficiency beside raw speed rather than as an afterthought. Its central result, that the first Tensor Processing Unit delivered inference performance 15 to 30 times that of the contemporary baselines in that study, on that study’s workloads, gains credibility precisely because the qualification travels with the figure. Stripped of workload and baseline, the same figure would assert far more than the evidence supports.

The discipline matters because purchasing, capacity planning and research comparison all rest on benchmark claims. A figure measured at a large batch size flatters throughput while hiding latency, the delay before a result arrives, which a small batch measurement would expose. A figure measured in a reduced precision format flatters speed while shifting the question of numerical accuracy onto the reader. A figure measured against an aging baseline flatters progress that a stronger baseline would shrink. Each omission is individually small, and together the omissions convert a measurement into an advertisement. The remedy is mechanical rather than moral. A claim that cannot state its workload, format, batch, configuration and baseline in one breath has not finished the measurement it reports, and a reader gains more from an incomplete table of honest conditions than from a single polished rate with no conditions attached at all.

Failure Modes: Heat, Faults and Numbers That Quietly Rot

An accelerator in service fails in several distinct ways, and only some of those failures announce themselves as a stopped machine. Sustained load raises die temperature until thermal throttling reduces clock rates, as the protective mechanism described in operating practice engages, and a training run that began at full speed settles into a slower rhythm without any error being recorded. Cooling degradation compounds the effect gradually. Dust in a heat sink, a failing fan or a blocked vent reduces heat removal over months of operation, the settled clock rate falls with it, and the system reports success on every job while quietly delivering less work per hour than its specification and its early service record promised.

Memory faults present a different character. An accelerator holds model weights, activations and intermediate results in vast arrays of dynamic random access memory, and individual cells in those arrays occasionally return a wrong bit through electrical disturbance or device wear. Error correcting code memory answers this fault class, storing extra check bits beside each word of data so that a single bit error is detected and corrected on reading, and a larger error is at least detected and reported rather than passed silently into a calculation. Without that protection, a flipped bit inside a weight changes the model itself, and the resulting errors appear as degraded output quality rather than as any recognisable hardware symptom.

Interconnect faults carry consequences beyond the device that suffers them. Large training runs divide one calculation across many accelerators that exchange partial results through a collective operation, a step in which every participant contributes data and every participant waits for the combined result. A fault that removes one device from that exchange stalls the whole group, because the remaining devices cannot complete the collective step with a missing contribution. The failure of a single cable, connector or device therefore idles hardware worth far more than the faulty component, and the cost of a fault scales with the size of the cooperating group rather than with the price of the part that failed.

Numerical failure is quieter still, because the hardware works exactly as designed while the numbers lose meaning. Reduced precision formats carry a narrow exponent range in the case of 16 bit floating point, which keeps 5 exponent bits, so a large intermediate value can overflow beyond representation and become infinite. At the other end of the scale, a small gradient, the signal that directs weight adjustment during training, can underflow below the smallest representable value and flush to zero, erasing the learning signal for that weight without any error flag. Neither event crashes a program. Both degrade the trained model in ways that surface later, during evaluation, separated in time and place from the arithmetic that caused them.

System level practice answers these failure classes with checkpointing, the periodic saving of complete model and optimizer state to durable storage during a long training run. When a fault interrupts the run, work resumes from the last saved checkpoint, and the loss is measured in minutes of repeated computation rather than in the weeks a restart from the beginning would cost. Checkpoint frequency itself is a trade, since writing state consumes time and storage bandwidth that computation could otherwise use, and operators balance that cost against the expected interval between faults. The combination of corrected memory, detected faults and regular checkpoints converts failure from a catastrophe into an operating expense, budgeted in advance and absorbed as routine.

Before the Accelerator: Vector Machines, Signal Processors and Graphics Chips

Specialized arithmetic hardware long predates machine learning, and the accelerator inherits ideas from three distinct ancestors. Vector supercomputers applied one instruction to long arrays of numbers in sequence, filling a pipeline of arithmetic units that produced a result on every cycle once the pipeline was full. That design contributed the vector idea itself, the recognition that regular data parallel work, in which the same operation repeats across independent values, rewards hardware built to stream operands rather than hardware built to juggle unrelated instructions. The vector machine paid for its arithmetic units with elaborate memory systems designed to keep the pipeline fed, establishing a pattern that every later design in this lineage repeats in some form.

Digital signal processors formed the second ancestor, built for multiply accumulate work in audio, telecommunications and radar processing. A signal processor filtering a stream of samples performs the same operation continuously, multiplying each sample by a coefficient and adding the product into a running sum, and its architecture reflected that discipline. Designers provided dedicated multiply accumulate units, memory arrangements that delivered a sample and a coefficient in the same cycle, and instruction forms shaped around loops that never changed shape. The contribution was the multiply accumulate discipline as an organizing principle, the willingness to shape an entire processor around one repeated operation because the workload guaranteed that the operation would dominate execution time.

Graphics processors formed the third ancestor, built to shade millions of pixels, each pixel requiring the same small calculation applied to independent data. Rendering a scene meant transforming vertices through matrix operations and evaluating lighting and texture for every point on a screen, a workload with enormous parallelism, meaning many independent calculations proceeding at once, and modest communication between them. Manufacturers answered with thousands of small floating point units arranged for throughput rather than for the low latency of any single calculation. No designer of those chips aimed at neural networks. The hardware existed to draw images, and its suitability for learning arithmetic was an accident of shared mathematical structure between the two workloads.

The hinge between that accidental suitability and deliberate use arrived with Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, whose paper ImageNet Classification with Deep Convolutional Neural Networks, presented at NeurIPS in 2012, trained the AlexNet model on two Nvidia GTX 580 graphics processors. Training on central processors would have taken far longer, and the graphics processors delivered the large matrix multiplications at the heart of the network at a rate that made the experiment practical. That result redirected attention across the field. Laboratories that lacked graphics hardware acquired it, software libraries for learning on graphics processors multiplied, and the economics of the graphics industry, which spread design costs across a large gaming market, gave researchers arithmetic capacity that no purpose built learning chip of the period could match in availability or price.

Each ancestor therefore left a distinct inheritance that the dedicated accelerator later combined. Vector machines contributed streaming pipelines and the memory systems that feed them. Signal processors contributed the elevation of one operation into an architectural center. Graphics processors contributed sheer numbers of floating point units, a manufacturing scale driven by an unrelated market, and a generation of programrs who understood how to express work as data parallel arithmetic. The purpose built accelerator did not replace that inheritance. It rearranged the same elements with the balance shifted, removing machinery for graphics and general control, enlarging the matrix units, and spending the recovered budget on memory bandwidth and on the data paths that keep dense arithmetic occupied.

Graphics Processors Against Purpose Built Designs

A comparison between a graphics processor and a purpose built accelerator earns its conclusion only when the criteria are stated in advance and applied in order. The first criterion is flexibility. A graphics processor runs irregular code, experimental operators and research prototypes without waiting for silicon, because its programmable units accept whatever instruction sequence a compiler emits. A researcher testing an unfamiliar layer shape or an unusual activation pattern can run the experiment on the day the idea forms. A purpose built design, shaped around expected operations, handles work outside its target set slowly or not at all, and the gap between the two widens exactly where research moves fastest and workload shapes remain unsettled.

The second criterion is efficiency. A purpose built design removes machinery its workload never uses, including graphics pipelines, elaborate cache hierarchies and control logic for irregular branching, and spends the recovered transistor and energy budget on matrix units and data movement. On the shapes it targets, that concentration wins energy per operation and operations per unit of silicon area. The graphics processor carries its general machinery as overhead on every calculation, paying area and power for capability that dense matrix work leaves idle. Mark Horowitz’s ISSCC 2014 energy accounting sharpens the point at system level, since energy follows data movement more than arithmetic, and a specialized data path that shortens movement extracts savings no arithmetic improvement alone can equal.

The third criterion is programmability. The graphics processor arrives with mature libraries of tuned operations, compilers refined through years of use, debugging tools and a large population of engineers who already write for it. A purpose built design typically arrives with a narrower toolchain, fewer tuned operations and a smaller community to answer questions when a model fails to compile or runs slowly for reasons the documentation leaves unclear. Programmability converts theoretical performance into delivered performance, because an operation missing from a library forces either a slow fallback or weeks of engineering, and either outcome erodes the efficiency advantage that justified specialized silicon in the first place.

The fourth criterion is risk. A model architecture change can strand fixed function hardware, leaving an expensive chip optimized for operations the field has moved beyond, while a flexible chip merely runs an unfamiliar shape less elegantly and remains useful. Silicon design cycles run to years, and the investment in a purpose built device must be recovered across a service life during which workloads will certainly shift in detail. Tensor cores, introduced in Nvidia’s Volta architecture in 2017, illustrate the hybrid answer to that risk, a specialized matrix unit absorbed into a flexible graphics processor, capturing much of the efficiency gain on dense operations while the surrounding programmable hardware absorbs whatever the matrix unit cannot handle.

The verdict follows from the criteria rather than from loyalty to either camp. The flexible design is the safer default for changing workloads, for research groups whose models shift monthly, and for any setting where engineering time costs more than electricity. The specialized design wins where the workload is large, stable and repeated, where one model shape runs millions of times and its energy bill dominates total cost. That is a decision rule rather than a tie. A buyer or architect who can describe the workload’s stability and scale can apply the rule directly, and a workload that fails the stability test should not be assigned to fixed silicon on the strength of a peak rate measured under conditions the workload will never reproduce.

The Case Against Specialization, Given Its Best Form

John L. Hennessy and David A. Patterson argued in Communications of the ACM in 2019 that domain specific architectures offer the path forward as general purpose scaling slows, and that argument deserves the strongest opposing evidence rather than a weak summary. Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang and Joel S. Emer, in Efficient Processing of Deep Neural Networks: A Tutorial and Survey, published in Proceedings of the IEEE in 2017, warned that flexibility and energy pull in opposite directions, so efficiency bought with fixed function is paid for in adaptability. Every operation hardened into silicon is an operation the device performs superbly, and every operation left out becomes a liability the moment a model requires it.

The historical record strengthens that warning considerably. Flexible graphics processors absorbed workload after workload across more than a decade of machine learning practice, beginning with the AlexNet training reported by Krizhevsky, Sutskever and Hinton in 2012 and extending through convolution, recurrent computation, attention and the rest of the field’s changing repertoire. When dense matrix multiplication grew dominant, the flexible platform absorbed even the matrix unit itself, as tensor cores entered the Volta architecture in 2017, gaining specialized efficiency without surrendering general programmability. A critic of specialization reads that record as evidence that the flexible platform captures most of the available gain on a delay of one product generation, while dedicated devices must win outright and immediately to justify their existence at all.

Timing deepens the objection. A specialized chip takes years to move from specification through design, verification and fabrication, while model architectures change within months, driven by published research that any laboratory can adopt on flexible hardware the week a paper appears. Silicon therefore risks arriving aimed at a departing workload, tuned for the operator mix, precision format and matrix shapes of a model family already falling from favor. The designer bets not only that dense arithmetic will dominate, which has held, but that particular shapes and formats will dominate, which has shifted repeatedly. Each shift devalues fixed decisions and rewards the programmable machinery that dedicated designs removed to pay for their efficiency advantage.

Fragmentation multiplies the cost on the software side. Every distinct architecture demands its own compiler work, its own tuned libraries, its own debugging tools and its own body of engineering expertise, and the field’s collective effort divides across however many platforms compete for adoption. A laboratory supporting several specialized devices repeats integration work that a single flexible platform would require once. Researchers face pressure to shape models around available hardware rather than around the problem, an inversion in which the tool constrains the science. The strongest form of the case concludes that specialization taxes the ecosystem continuously, in engineering hours and foregone experiments, while its energy savings arrive only on workloads that remain stable long enough to collect them.

The thesis therefore requires honest narrowing in response to that case. Specialization wins at the level of units inside flexible chips, where a matrix engine sits beside programmable cores that absorb change, and it wins at the largest stable workloads, where one shape repeated at enormous scale repays fixed design many times over. Specialization loses as a bet that one whole chip shape will match a field that keeps moving, because the field’s movement is documented, continuous and faster than fabrication. That narrowed claim survives the opposing evidence. The broader claim, that dedicated silicon should displace flexible hardware across machine learning generally, does not survive it, and retaining the narrower claim makes the argument stronger rather than weaker because its boundaries match the record.

Software Is the Moat That Hardware Cannot Print

An accelerator reaches a laboratory as silicon, yet the laboratory experiences it as software. Libraries of tuned operations supply the matrix multiplications, convolutions and normalization routines that models call constantly, each routine adjusted by hand or by automated search for particular shapes and memory layouts. Compilers translate a model description into sequences of those operations, allocate memory, schedule transfers and fuse small operations into larger ones to reduce movement. Debuggers and profilers reveal where time actually goes when delivered performance falls short of a specification. Trained engineers carry knowledge of all three layers, and a decade of accumulated fixes inside that stack records solutions to failures that a younger platform has not yet encountered, let alone repaired.

The mechanism behind platform durability is the switching cost created by the programming model, the set of tools and interfaces through which a chip is programd. Code written against one vendor’s programming model expresses parallelism, memory placement and synchronization in that model’s terms, and those terms do not translate mechanically into another model’s terms. A port therefore means rewriting performance sensitive sections, retesting numerical behavior, and retraining engineers whose habits and diagnostic instincts belong to the original platform. Each individual barrier is surmountable. Together they compound into a cost that a hardware advantage must exceed before migration becomes rational, and a challenger that is faster on paper by a modest margin rarely clears the accumulated total.

Tuned libraries deepen the lock through a subtler route. A model spends most of its execution inside a small number of library routines, so the practical performance of a platform is largely the performance of its libraries rather than of its raw arithmetic units. Building a library that matches a mature competitor’s requires years of shape by shape optimization, testing across model families, and correction of edge cases that appear only under particular dimensions or data layouts. That work cannot be hurried by transistor budgets, because it consists of engineering attention applied sequentially to thousands of details. Hardware can be fabricated on a schedule. A library’s accumulated coverage cannot, and the difference explains why platform contests resolve slowly even when silicon specifications suggest a rapid outcome.

Engineers themselves form part of the moat, through knowledge that lives in hiring pools and institutional memory. A laboratory recruiting for a mature platform draws on many candidates who have debugged its failure modes before, while a laboratory adopting a younger platform trains its own specialists and accepts slower progress during that training. Published answers to common errors, course materials, example code and community discussion all accumulate around the platform with the larger installed population, and each addition lowers the cost of staying while raising the relative cost of leaving. None of this appears in a comparison of arithmetic rates or memory bandwidth, yet it decides which hardware a laboratory can actually use at full capability within a project’s lifetime.

Judged by that mechanism, durability belongs to whichever platform holds the deepest software position, almost regardless of which device posts the stronger isolated measurement in a given generation. Challengers answer by investing in translation layers, open programming interfaces and library coverage, an acknowledgement that the contest runs through software rather than around it. Incumbents defend by extending their stacks into adjacent tasks, so that data preparation, deployment and monitoring also accumulate inside one environment. The arithmetic facts of a chip set the terms on which it may compete. The software surrounding the chip determines whether that competition happens at all, and any account of accelerator markets that begins with silicon and ends with silicon has omitted the layer where outcomes are usually settled.

How a Practitioner Reads an Accelerator Claim

A practitioner reading an accelerator claim begins by asking which workload produced the headline figure, and treats any answer that names a synthetic test rather than a recognisable model as a reason to discount the result. The model, its size and its operator mix determine whether arithmetic units stay occupied or wait on memory, so a claim transfers to a different workload only when the two share shape and structure. Attention then moves to the number format, because a rate measured in a reduced precision format describes different arithmetic from the same rate measured at higher precision, and the comparison figures belong in separate columns until the format question is settled for each entry under consideration.

Memory receives equal scrutiny in that reading, since a practitioner knows the arithmetic rate is rarely the binding limit in delivered work. Capacity decides whether a model and its working data fit on the device at all, and bandwidth, the amount of data moved per second, decides how fast the arithmetic units can be fed once the model fits. Batch size completes the picture, because a large batch raises throughput while extending the wait for any individual result, and a claim measured at a batch no deployment would use describes a configuration rather than a capability. If you compare two claims, you will find the honest comparison begins only after workload, format, batch, capacity and bandwidth have been matched line by line.

Compiler maturity draws the next question, framed around how much of the promised performance a real model can reach without specialist intervention. A practitioner asks which operations the compiler handles natively, which fall back to slower paths, and how the tooling reports the difference, because a model that compiles with silent fallbacks delivers a fraction of its expected rate for reasons invisible in summary results. Energy closes the examination, measured per finished job rather than per operation, since a device that performs each operation cheaply but requires extensive data movement or long runtimes can lose the energy comparison at the level that determines operating cost. Your final test is therefore practical rather than theoretical: a claim survives when its conditions resemble the intended workload closely enough that the measured result predicts the delivered one.

Duration and scale receive the same treatment in a careful reading. A practitioner asks how long a measurement ran, because a rate held for seconds can fall once thermal limits engage and clock frequencies settle at the level cooling sustains. A practitioner also asks how the result changes as devices are added, since a single device working alone avoids the collective exchanges that slow a cooperating group, and a claim measured on one device predicts a multi device deployment poorly when interconnect transfers dominate the schedule. The related question concerns what happens when a model exceeds device memory, because spilling data to slower memory outside the device changes the feeding pattern entirely and can reduce delivered performance by a factor that no arithmetic specification anticipates or discloses.

A practitioner may keep those figures together in a companion calculation ledger for keeping the comparison figures in one place.

The Limit Moves and the Design Follows It

The history of the accelerator is the history of a bottleneck moving, and each design generation records a wager on where that bottleneck sits. Arithmetic formed the first limit, when central processors performed matrix operations too slowly for models of useful size, and the earliest answer multiplied arithmetic units without restraint. That answer exposed the second limit, memory, because thousands of units consumed operands faster than memory systems could supply them, and designs responded with wider interfaces, stacked memory and on-chip storage that held tiles close to the units. Performance came to be judged by feeding rather than by counting, and the roofline model described by Samuel Williams, Andrew Waterman and David Patterson in Communications of the ACM in 2009 gave that judgement its standard diagram.

Movement of the limit continued outward. Once single devices fed their units adequately, training at scale bound many devices together, and interconnect became the third limit, the speed at which partial results move between devices during collective operations. A group of accelerators computes at the pace its slowest exchange permits, so link bandwidth and the topology connecting devices entered design calculations that arithmetic alone once governed. Designs answered with dedicated high speed links between neighbouring devices and with communication scheduled to overlap computation, hiding transfer time behind arithmetic wherever the structure of the calculation allowed the two to proceed together rather than in strict alternation.

Energy became the fourth limit, binding in facilities where power delivery and cooling capacity are fixed by the building rather than by ambition. Mark Horowitz’s ISSCC 2014 accounting, in which a memory access cost hundreds of picojoules against a few picojoules for a multiplication at 45 nanometres, explains why that limit arrived and why it persists. Arithmetic grew cheap through decades of process improvement, while moving data retained costs set by distance, capacitance and the physics of signalling. A facility can purchase additional accelerators in an afternoon of procurement, yet it cannot purchase additional megawatts for a hall whose electrical service and cooling plant were sized years earlier, so energy per finished job governs how much computation a site can host.

One lesson survives every migration of the limit, and a reader carries it into every future claim. Judge every accelerator by what it feeds its arithmetic units and what it gives up to do so, because the specification that headlines a product describes the units themselves while the delivered result depends on the feeding system, and the surrendered generality determines whether the device remains useful when workloads shift. Designs that feed superbly and surrender narrowly have aged well across generations of models. Designs that fed poorly wasted their arithmetic regardless of its quantity, and designs that surrendered broadly became stranded when the field moved. That single test, applied patiently, separates durable engineering from specification theatre in generation after generation.

The structural implication extends beyond any single device family or vendor. As long as machine learning remains regular arithmetic performed at vast scale, silicon will keep specializing around that regularity, adding units for whichever operation dominates and data paths for whichever movement constrains it. The contest between designs will keep being decided by movement rather than multiplication, because multiplication is the part engineering made cheap while movement retains the costs that physics imposes. Bottlenecks will relocate as each generation relieves the binding one, and the designs that endure will be those whose architects identified the relocation early and spent their budgets at the point where the limit was arriving, not where it had been.

Frequently Asked Questions

Q: What is an AI accelerator in simple terms?

An AI accelerator is a computer chip built for one family of calculations, the matrix and vector arithmetic at the heart of machine learning. A general processor divides its attention among many kinds of programs, while an accelerator dedicates most of its silicon to thousands of small units that each perform a multiply accumulate, multiplying two numbers and adding the product to a running total. Around those units sit wide paths to memory and explicitly managed on-chip storage, arranged so the units rarely wait for data. The design gains speed and energy efficiency by surrendering versatility. Software outside its narrow family runs slowly or not at all, and that surrender is the source of its advantage rather than a side effect of it.

Q: How is an AI accelerator different from a normal processor?

A normal processor is built for irregular programs whose next step depends on results that arrive unpredictably, so it spends much of its silicon on branch prediction, out of order execution, speculative work and large caches. Only a fraction of its transistors perform arithmetic in any given cycle, and its goal is low latency, the delay before one result arrives, on a single instruction stream. An accelerator removes most of that control machinery because its workload follows a schedule known in advance. The freed silicon becomes dense arrays of simple arithmetic units sharing one set of control signals, plus wider memory paths to feed them. The difference is one of budget allocation. Each chip spends its area and energy on the problem it expects, and each performs poorly when handed the other’s problem.

Q: Why do AI accelerators contain so many small arithmetic units?

Matrix multiplication, the dominant calculation in machine learning, consists of enormous numbers of independent multiply accumulate operations, and independence allows the operations to proceed simultaneously rather than in sequence. Small units make simultaneity affordable. A simple multiplier and adder pair occupies little silicon, consumes modest energy, and needs no individual control machinery when thousands of neighbours share the same signals. Adding units raises throughput, the volume of work completed per second, at the existing clock rate, while raising the clock rate instead would climb steeply in energy and heat. The count of units is therefore the accelerator’s main route to speed. Its cost appears elsewhere, since every added unit is another consumer the memory system must feed, and an array that outgrows its feeding paths gains nothing from its extra width.

Q: What is a tensor core and what does it do?

A tensor core is a matrix unit built into a graphics processor, introduced in Nvidia’s Volta architecture in 2017. Where an ordinary arithmetic unit handles one pair of numbers per operation, a tensor core accepts small matrices as its operands and returns a matrix as its result, performing a whole block of multiply accumulate operations as a single instruction. That block operation matches the shape of neural network calculation, in which layers multiply matrices by matrices and vectors throughout. Placing the function inside a programmable graphics processor, rather than in a standalone specialized chip, keeps the surrounding flexibility of the host device while accelerating the dense arithmetic that dominates training and inference time. The tensor core shows that specialization can be added as one organ inside a general machine, not only as the entire machine.

Q: Why does lowering number precision make an accelerator faster?

Number precision is set by how many bits represent each value, divided between an exponent that fixes the range of magnitudes and a mantissa that fixes the fineness of steps within that range. Lower precision, such as bfloat16 with its 8 exponent bits and 7 mantissa bits, shrinks everything downstream. Multipliers for narrower values occupy less silicon, so more arithmetic units fit on one chip. Each operation moves fewer bytes between memory and the array, so a fixed memory bandwidth supports more operations per second. Energy falls on both counts, since the multiplication costs less and the data travel behind it shrinks. Machine learning tolerates the coarser representation because weights and activations are statistical quantities whose small errors tend to average out across millions of operations, and training protects its weight updates by keeping a high precision master copy.

Q: What is high bandwidth memory and why does it matter for AI chips?

High bandwidth memory is a memory technology in which dynamic memory dies are stacked vertically in a tower and connected to a processor through a silicon interposer, a dense wiring layer, over a very wide interface. The arrangement is standardized by JEDEC, the industry standards body. Width substitutes for connection speed, and the short distance between the stack and the processor keeps the energy per transferred bit low. The technology matters for AI chips because accelerator arrays consume data at rates that commodity memory interfaces, designed for central processors with modest streaming needs, cannot sustain. An array starved of data performs at its memory ceiling regardless of its arithmetic capacity, so the wide channel between the reservoir of weights and activations and the arithmetic engine decides how much of the chip’s rated performance a real workload can reach.

Q: What does the roofline model show about a processor?

The roofline model, introduced by Samuel Williams, Andrew Waterman and David Patterson in Communications of the ACM in 2009, shows which of two ceilings limits a workload on a given machine. One ceiling is the peak arithmetic rate, the most operations per second the hardware can perform under any conditions. The other rises with arithmetic intensity, the operations performed per byte moved from memory, and equals memory bandwidth multiplied by that intensity. Attainable performance is whichever ceiling is lower. A workload with high reuse of each loaded value reaches the arithmetic ceiling, while a workload that touches each value only once or twice is capped by bandwidth no matter how many multipliers the chip carries. The model turns arguments about speed into a measurable question about balance between computation and data movement.

Q: How does a systolic array work step by step?

A systolic array passes data rhythmically through a grid of small processing elements, each holding one weight in place while inputs flow past in one direction and partial sums flow in another direction or accumulate locally. An input value enters an element, multiplies against the stored weight, adds to a passing or resident total, and moves onward to the neighbouring element for multiplication against a different weight. Because each value serves many elements during a single journey across the grid, a matrix multiplication completes with far less traffic to main memory than a design that fetches a fresh pair of operands for every operation. Timing follows a regular beat set by the movement pattern, and the regularity allows the array to sustain high utilization when matrix dimensions match the grid dimensions well.

Q: Why does batching make inference more efficient and slower to answer?

Batching raises efficiency because one reading of the model weights serves every request in the group, dividing the movement cost of the weights across many answers and raising the operations performed per byte fetched. Arithmetic units that would idle during a single request gain parallel work from companion requests, so a larger share of hardware capacity becomes completed answers per unit of time and per unit of energy. The delay cost follows from grouping itself. A request waits while the group forms, waits while grouped execution runs, and finishes only with the group rather than at the moment individual calculation could have completed. Larger groups deepen both effects at once, improving total throughput while extending and varying the wait experienced by each person whose request joined the group.

Q: What is sparsity in a neural network calculation?

Sparsity is the share of zero values among the weights of a network or among the activations produced during a calculation. A zero weight means a connection contributes nothing, and a zero activation means a value contributes nothing to the next layer, so operations involving those zeros could in principle be skipped without changing the result. Zeros appear because pruning during training removes weak connections, and because certain activation functions return zero for negative inputs on every pass. Whether skipping saves time depends on pattern and hardware. Scattered zeros require indexing and irregular fetching that can erase the saving, while zeros arranged in whole rows or regular blocks let hardware omit complete sections of work under a predictable schedule.

Q: How do several accelerators share one large model?

Several accelerators share a large model by dividing the model itself across devices, with different layers assigned to different chips or with single large layers split into slices held on different chips. During a forward pass, activations produced on one chip travel across the connecting links to the chip holding the next layer or the matching slice, and during training the error signals travel back along the same routes in reverse. Partial products from sliced layers must be combined before a layer output is complete, which places communication inside every pass. The arrangement lets combined memory hold a model that exceeds any single device, at the cost of making link speed and division placement central to performance, since every chip waits on values in transit.

Q: What is a dataflow in accelerator design?

A dataflow is the pattern that determines where data rests and where data moves during a calculation on an accelerator. The pattern specifies which values stay inside a processing element for repeated use, which values stream through an array of elements, and where partial sums accumulate before completion. Different dataflows applied to identical arithmetic units produce different traffic to main memory, different energy costs, and different utilization from the same hardware. Row stationary mapping, for example, keeps a row of each operand at each element so both operands gain reuse where each sits. Designers choose a dataflow to match the reuse structure of the target workload, because movement rather than arithmetic dominates energy and time once arrays grow large.

Q: How does the compiler change how fast an accelerator runs?

The compiler changes delivered speed by deciding tile sizes, memory layout, operation fusion, and transfer scheduling before execution begins. Tile sizes determine how often weights and inputs are fetched again from main memory. Layout decisions determine whether values arrive at arithmetic units in the order of consumption or require costly gathering. Fusion determines whether intermediate results pass directly between operations on chip or make round trips to memory between separate passes. Scheduling determines whether transfers for future work overlap with arithmetic on present work or leave units idle while data arrives. Identical hardware under a weak plan spends much capacity waiting and refetching, while identical hardware under a strong plan keeps units supplied, so delivered performance differs severalfold through translation choices alone.

Q: Why does a model need to be loaded from memory for every generated token?

Autoregressive generation produces output one token at a time, and each token extends the input that the model must process to produce the next token. A further pass through the model therefore follows every token, and each pass needs the full set of model weights to compute its result. Weights that fit in fast on-chip storage can stay resident between passes, but a large model exceeds on-chip capacity, so most weights must be read again from main memory for each step. Hundreds of generated tokens mean hundreds of readings of the weight set. Memory bandwidth consequently governs generation speed, because the rate of producing tokens cannot exceed the rate at which complete passes over the weights can be supplied with data.

Q: Why did graphics processors become the first widely used AI accelerators?

Graphics processors became widely used for learning because their original purpose, rendering images, required the same mathematical pattern that neural networks require. Shading millions of pixels meant applying one small calculation to vast numbers of independent values, so manufacturers built thousands of floating point units arranged for throughput. Neural network training rests on matrix multiplication, which shares that structure of regular, parallel arithmetic. Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton demonstrated the fit at NeurIPS in 2012 by training AlexNet on two Nvidia GTX 580 graphics processors, achieving a practical training time that central processors could not approach. Availability completed the argument, since a large gaming market had already spread the hardware and its cost widely.

Q: What limits an AI accelerator besides its arithmetic rate?

Memory bandwidth limits an accelerator whenever its arithmetic units consume operands faster than the memory system supplies them, leaving expensive units idle between deliveries. Memory capacity sets a prior limit, because a model and its working data must fit on the device before any calculation begins. Interconnect speed limits groups of devices that exchange partial results during large training runs, since every participant waits for the slowest exchange in a collective step. Energy and cooling impose a further limit in sustained operation, as thermal protection reduces clock rates when heat removal falls behind heat generation. Compiler quality and library coverage limit delivered performance as well, because operations without tuned implementations run slowly regardless of the underlying silicon capability available.

Q: How accurate are low precision calculations for machine learning?

Low precision calculations trade numerical accuracy for speed, range and reduced data movement, and machine learning tolerates that trade better than many fields because trained models absorb small errors through their own adjustment process. Bfloat16 keeps the 8 exponent bits of 32 bit floating point while cutting the mantissa to 7 bits, preserving the representable range while accepting coarser values within it, which suits training where large and small magnitudes both appear. Sixteen bit floating point keeps 5 exponent bits and 10 mantissa bits, offering finer values across a narrower range, with overflow and underflow as the corresponding hazards. Accuracy in practice depends on where reduction is applied, since accumulated sums and weight updates usually demand more care than individual multiplications.

Q: Why do accelerators slow down when they run hot for a long time?

Accelerators slow under sustained heat because temperature sensors trigger a protective reduction in supply voltage and clock frequency once the die approaches its thermal limit, a response called thermal throttling. Heat generation rises with operating rate, and a cooling system removes heat at a rate fixed by its design, airflow and the temperature of the surrounding room. A brief workload finishes before temperatures climb, so short measurements record full speed, while a long workload reaches the point where generation and removal balance, and the device settles at whatever rate that balance supports. Dust accumulation, failing fans and obstructed vents lower the removal rate gradually, so the settled speed of an aging installation can fall below its earlier performance without any component failing outright.

Q: What makes one accelerator better for training and another better for inference?

Training and inference place different demands on hardware because training performs far more work per input. Backpropagation adjusts weights by propagating an error signal backwards through the network using the chain rule, so training involves roughly three passes of matrix work, covering the forward pass, the backward pass for activations and the backward pass for weights, against a single forward pass for inference. Training therefore rewards large memory capacity for weights, gradients and optimizer state, high bandwidth, and fast links between cooperating devices. Inference, meaning the use of a trained model to produce an output, rewards low latency for individual requests, energy efficiency at modest batch sizes, and predictable response under varied load, so a device optimized for either task sacrifices qualities the other values.

Q: How should a benchmark result for an AI accelerator be read?

A benchmark result deserves reading as a claim about specific conditions rather than as a general property of the hardware. The conditions to identify are the workload and model tested, the size of that model, the number format used, the batch size, the hardware configuration and the baseline chosen for comparison. Norman P. Jouppi and co-authors modeled that discipline at ISCA in 2017, naming workloads and baselines and reporting energy beside speed, which allowed readers to judge how far results might transfer. A result missing those conditions cannot be checked or applied to a different setting with confidence. Peak rates in particular describe arithmetic units under perfect feeding, so delivered performance on real workloads, with real memory limits and compiler behavior, remains the figure that matters.