ResearchAudio.io

Hardware, in depth

65,536 Multipliers, One Clock Cycle

Inside Google's TPU: the systolic array that made specialized silicon win

The accelerator behind Google's Gemini looks nothing like the GPU most engineers picture. It has a tiny cache, handles branching poorly, and would lose a general-purpose coding contest to a laptop chip. On the one operation that dominates a neural network, the dense matrix multiply, it runs tens of thousands of multipliers in lockstep and barely touches memory in between.

That chip is the Tensor Processing Unit, Google's custom silicon for machine learning. This is a look at how it actually works: the systolic array at its heart, the bfloat16 math it runs, why memory is the wall you hit first, and how nine thousand of these chips get wired into one machine.

Why doing less lets a chip do more

Training or serving a transformer is, at the arithmetic level, mostly one thing: multiplying large matrices. Each layer runs the query, key, value, and output projections, then two feed-forward layers, and every one of those is a dense matmul. A general-purpose chip spends most of its transistors on flexibility (branch prediction, speculative execution, deep caches) that a matmul does not need. Google's wager, made in the mid-2010s, was that a chip stripped down to matrix math would win on performance per watt for exactly this workload.

The deeper constraint is not multiply capacity. It is moving data. In a conventional processor, every multiply-accumulate reads two operands from a register file or cache and writes the result back. Those reads and writes, not the arithmetic, dominate the energy and the clock time. This is the memory wall, and it grows worse as models get larger.

Inside the systolic array

The heart of a TPU is the Matrix Multiply Unit (MXU), a two-dimensional grid of multiply-accumulate cells. Think of it as an assembly line: each worker performs one small step and hands the part to the next, instead of everyone walking back to a shared warehouse after every step. Weights are loaded once into the grid and held in place. Activations enter from the left edge and march across, one column of cells per clock cycle.

Each cell multiplies its stored weight by the incoming activation, adds the partial sum arriving from the cell above, and passes the new partial sum downward. After the data ripples through, each column at the bottom holds a finished dot product. Because operands stay in place and partial sums move cell to cell, the array runs thousands of multiply-accumulates without a single trip to HBM or a register file. That is the whole trick: the expensive part, memory access, is amortized across a huge grid of inexpensive arithmetic.

On TPU v2 through v5p, that grid measured 128 by 128, which is 16,384 cells firing every cycle. Trillium (v6e) doubled each side to 256 by 256, or 65,536 cells, quadrupling the work per cycle. This geometry reaches into your code. The array processes fixed-size tiles, so your matrix dimensions should be multiples of 128, or 256 on the newer chips. A matrix that is 130 wide gets padded up to the next tile, and most of that tile then runs empty, wasting throughput you already paid for.

bfloat16, the format built for training

TPUs run their matmuls in bfloat16, a 16-bit number format Google designed for deep learning. It keeps the wide exponent range of a 32-bit float and drops precision bits the training process can tolerate losing. Concretely, bfloat16 uses the same 8 exponent bits as fp32 but keeps 7 mantissa bits instead of 23. The MXU takes bfloat16 inputs and accumulates the running sum in fp32, so the total stays accurate even though each input is coarse.

The payoff is stability. The full-width exponent means gradients rarely underflow, so teams often skip the loss-scaling workarounds that half-precision needs on other hardware. Ironwood goes further and adds native FP8 for inference, trading precision for throughput at serving time, where a little numerical noise costs less.

Memory is the wall you hit first

A TPU chip has three tiers of memory, and data has to march inward to be useful. Farthest out is HBM, the high-bandwidth memory stacked next to the chip. Ironwood carries 192 GB of HBM3e at roughly 7.4 TB/s, up from 32 GB and 1.6 TB/s on Trillium. Closer in is VMEM, a small on-chip scratchpad with far higher bandwidth to the MXU, where custom kernels stage data to keep the array fed.

Here is the part that bites in practice: capacity, not raw compute, usually decides what you can run. A 175-billion-parameter model in bfloat16 needs about 350 GB for weights alone, more than a single Ironwood chip's 192 GB, before you count activations or the KV cache. That is why large models are sharded across many chips, and why the HBM number on the spec sheet often matters more than the FLOPS.

Wiring nine thousand chips into one machine

One chip is rarely enough, so TPUs connect straight to each other over an inter-chip interconnect, the ICI, without routing through a host CPU. Ironwood gives each chip 1.2 TB/s of ICI bandwidth. The chips are wired into a torus, a grid that wraps around at the edges so no chip sits on a lonely boundary with fewer links.

Cost-tuned chips (v5e, v6e) use a 2D torus that scales to 256 chips per pod. Performance chips (v4, v5p, Ironwood) use a 3D torus, where each chip links to six neighbors, and scale much larger: a v5p pod holds 8,960 chips, an Ironwood pod 9,216. Since v4, an optical circuit switch sits between racks and can physically rewire the topology in seconds, routing around a failed chip or reshaping the mesh to fit a job. Beyond one pod, Google's Pathways runtime stitches many pods together.

The scale is physical. A full Ironwood pod of 9,216 liquid-cooled chips draws close to 10 megawatts, roughly a thousand watts per chip. This is the hardware a growing share of frontier training and serving runs on.

One idea, five generations

The first TPU was running inside Google's data centers by 2015 and shown publicly at Google I/O 2016. It was a 256 by 256 array of 8-bit multipliers built for inference alone, and it could not train a model. The second generation added training and bfloat16, and shrank the array to 128 by 128. Trillium later went back to the original 256 by 256 size, so the newest chips echo the very first one.

Specs from Google Cloud TPU documentation and public announcements. *Ironwood BF16 is an estimate; its native FP8 peak is 4,614 TFLOPS.

Ironwood, the seventh generation, marks a shift. Announced at Google Cloud Next in April 2025 and commercially available since November 2025, it is the first TPU designed for inference rather than training. The logic is plain economics: training a frontier model is a one-time cost, but serving it to millions of users runs every hour and, over a model's life, dwarfs the training bill many times over.

The software you actually touch

You almost never program the MXU by hand. You write JAX, TensorFlow, or PyTorch (through PyTorch/XLA), and the XLA compiler lowers your computation graph to TPU instructions. XLA does the hard parts: tiling matrices to the array, spreading tensors across HBM channels, and fusing operations to cut memory traffic. When you need finer control, Pallas lets you write custom kernels against VMEM directly.

There is a catch that echoes the hardware. XLA rewards graphs with static, regular shapes and punishes dynamic shapes and heavy branching, the same patterns the systolic array dislikes. Code that feels lightweight in Python, a data-dependent loop or a variable-length sequence, can quietly serialize the array and stall the pipeline.

TPU versus GPU, the tradeoff

The contrast with a GPU is clean. A GPU spreads work across thousands of small, flexible cores that handle irregular compute well, which is why it stays a strong generalist. A TPU pours its silicon into one or a few large systolic arrays that do dense matmul with very little waste but handle irregular work poorly. A GPU is a generalist; a TPU is a specialist that wins when the workload is mostly big dense matrix multiplies.

That specialization cuts both ways, and it may help explain why transformers took over. Their compute is almost entirely large dense matmuls, which map cleanly onto hardware like this. The architecture that scaled best was, in part, the one that fit the silicon.

What this means for your work

A TPU is a bet that most of modern AI is one operation run at enormous scale, and that spending the entire chip on that operation beats keeping it flexible. Every choice follows from the wager: the stationary weights, the 256-wide tiles, the torus of 9,216, the compiler that wants your graph to hold still.