Architecture

A CPU is a latency-optimized processor. It is designed to execute a single thread of instructions as fast as possible. To achieve this, modern CPUs dedicate enormous amounts of transistor budget to machinery that has nothing to do with arithmetic: branch predictors, out-of-order execution engines, register renaming, speculative execution, multi-level cache hierarchies (L1/L2/L3), and memory prefetchers. On a modern Intel or AMD CPU, the actual arithmetic units—the ALUs and FPUs—occupy perhaps 5–10% of the die area. The rest is control logic and cache.

This makes the CPU extraordinarily flexible. It can evaluate an if/else branch with near-zero penalty (thanks to branch prediction). It can chase pointers through a linked list. It can handle interrupts, context-switch between threads, run an operating system kernel, parse a file, manage a network stack, and execute arbitrary code with complex data dependencies between operations. No other processor type can do this.

The cost of this generality is that the CPU has relatively few arithmetic units. A high-end desktop CPU in 2024 has 8–24 cores, each capable of executing perhaps 32–64 FP32 operations per cycle (via SIMD/AVX instructions). That gives it a few teraflops at peak—impressive by historical standards, but a fraction of what a GPU achieves.

The SIMD Revolution

CPUs have steadily widened their arithmetic throughput through SIMD (Single Instruction, Multiple Data) extensions. Instead of operating on one number at a time, SIMD instructions operate on a vector of numbers in a single cycle:

• SSE (1999): 128-bit registers. 4 FP32 or 2 FP64 operations per instruction.
• AVX (2011): 256-bit registers. 8 FP32 or 4 FP64 operations per instruction.
• AVX-512 (2016): 512-bit registers. 16 FP32 or 8 FP64 operations per instruction.

With fused multiply-add (FMA), each SIMD instruction counts as two operations (a multiply and an add), doubling the effective throughput. A single AVX-512 FMA instruction on FP32 data performs 32 floating-point operations in one cycle. But you need data that is independent and contiguous to fill these wide registers. Code with branches, pointer chasing, or irregular memory access patterns cannot use SIMD effectively. This is the fundamental constraint: the CPU is fast at everything, but only achieves peak FLOPS on embarrassingly parallel, regular data.

Historical FLOPS: CPU

From 50 thousand to 2.9 trillion in 42 years: a factor of 58 million. But notice the trajectory flattening. The jump from the 8087 to the Pentium III (1980–1999) was $80{,}000\times$. The jump from the Pentium III to the Ryzen 9 7950X (1999–2022) was about $725\times$. Clock speeds plateaued around 2005 (the "power wall"); further gains came from more cores and wider SIMD, which only help if the workload is parallel.

What CPUs Are Best At

Operating systems. Compilers. Databases. Web servers. Game logic. Any code with complex branching, irregular memory access, or serial dependencies between operations. The CPU is the only processor that can run arbitrary programs. Everything else on this list is a specialist.

Architecture

A GPU is a throughput-optimized processor. Where a CPU dedicates transistors to making one thread fast, a GPU dedicates transistors to running thousands of threads simultaneously. An NVIDIA H100 has 16,896 CUDA cores, compared to 24 cores on a top-end Intel CPU. Each GPU core is far simpler than a CPU core: no branch predictor, no out-of-order execution, minimal cache. But there are thousands of them, and they all execute the same instruction at the same time on different pieces of data (SIMT: Single Instruction, Multiple Threads).

This architecture emerged from the demands of real-time graphics. Rendering a frame requires performing the same computation (lighting, shading, texture sampling) on millions of pixels independently. There are no dependencies between pixels: the color of pixel (300, 400) does not depend on the color of pixel (301, 400). This is embarrassingly parallel, and the GPU is built for exactly this pattern.

The Constraint: No Dependencies

The GPU achieves its enormous FLOPS counts precisely because it assumes operations are independent. If your computation requires the result of step N before it can begin step N+1, the GPU cannot help you. Thousands of cores will sit idle waiting for a single dependency to resolve. Consider these two examples:

• Matrix multiplication (GPU-friendly): Every element of the output matrix can be computed independently. Feed the GPU two $4096\times4096$ matrices and it will compute all 16 million dot products in parallel. The H100 sustains close to its peak tensor FLOPS on this workload.
• Fibonacci sequence (GPU-hostile): $F(n) = F(n-1) + F(n-2)$. Each term depends on the previous two. This is inherently serial. A GPU computes this slower than a CPU because of the overhead of launching a kernel for a single addition.

Real workloads fall on a spectrum between these extremes. Neural network training is dominated by matrix multiplications (GPU-friendly). Game physics involves both parallel particle systems (GPU-friendly) and sequential constraint solving (GPU-hostile). Sorting algorithms are partially parallelizable but require synchronization. The art of GPU programming is restructuring algorithms to minimize dependencies and maximize parallel occupancy.

Tensor Cores and the Precision Hierarchy

Starting with NVIDIA’s Volta architecture in 2017, GPUs added tensor cores: specialized matrix-multiply units that operate on small matrices (e.g., $4\times4$) in a single cycle. Tensor cores exploit the fact that deep learning tolerates reduced precision. The result is a dramatic hierarchy of throughput by precision:

Look at that table carefully. The B200’s FP4 tensor throughput (18,000 TFLOPS) is $486\times$ its FP64 throughput (37 TFLOPS). This is not a software trick. At lower precision, the arithmetic units are physically smaller, so more fit on the die, and each completes faster. The chip is literally doing $486\times$ more operations per second—but each operation carries only 4 bits of precision instead of 64. For scientific simulation (climate models, fluid dynamics, molecular dynamics), you need FP64. For training large language models, FP16 or BF16 is standard. For inference, FP8 or even FP4 is increasingly sufficient.

The "with sparsity" column refers to NVIDIA’s 2:4 structured sparsity feature (Ampere and later): if exactly 2 of every 4 values in a matrix are zero, the hardware skips those multiplications, effectively doubling throughput. This is a hardware-level exploitation of the fact that neural network weights are often sparse.

Historical FLOPS: GPU

From the GeForce 256 to the B200 in 25 years: FP32 went from ~2 GFLOPS to 75 TFLOPS ($37{,}500\times$). But the real story is the AI-specific formats. The V100 in 2017 introduced tensor cores at 125 TFLOPS FP16. Seven years later, the B200 reaches 18,000 TFLOPS at FP4—a $144\times$ increase in AI throughput in seven years, on top of the preceding two decades of GPU evolution.

What GPUs Cannot Do

A GPU cannot efficiently run an operating system. It cannot handle unpredictable branches. It cannot chase pointers through a graph. It cannot serve a web request. It cannot do anything that requires one operation to complete before the next can begin. The thousands of cores are powerful only when they all do the same thing at the same time. When threads within a warp (a group of 32 threads on NVIDIA hardware) take different branches, the GPU serializes both paths—a penalty called warp divergence. The GPU also has limited cache and relies on massive parallelism to hide memory latency (by switching to another warp while one waits for data). If your workload has irregular memory access patterns, the GPU stalls.

The GPU is a specialist that happens to be useful for an enormous category of problems: anything that looks like "apply the same function to millions of independent data points." Graphics, matrix algebra, neural networks, physics simulations, signal processing, cryptography, and Monte Carlo methods all fit this description. But the CPU remains indispensable for everything else.

Architecture

Neural networks, at the level of hardware, are almost entirely matrix multiplications. A transformer layer is a sequence of matrix multiplies (queries $\times$ keys, attention $\times$ values, feed-forward projections) with nonlinear activations (softmax, GELU) between them. The activations are elementwise and cheap. The matrix multiplies dominate runtime. An NPU or TPU is built around a systolic array: a grid of multiply-accumulate (MAC) units that data flows through in a wave, performing matrix multiplication with maximum arithmetic density and minimal data movement.

Google’s TPU v1 (2016) had a $256\times256$ systolic array of 8-bit integer MACs—65,536 multipliers operating in lockstep. It could not train models (no backpropagation support); it was inference-only. But it could evaluate neural networks at 92 trillion 8-bit operations per second. Each subsequent generation added training capability, higher precision (BF16), more memory, and faster interconnects.

The key difference from a GPU: an NPU/TPU has no general-purpose cores. There is no CUDA-equivalent programming model where you write arbitrary kernels. The chip executes a fixed computational graph (a neural network) that has been compiled into a sequence of matrix operations. This rigidity is the source of its efficiency. By not supporting arbitrary computation, the chip dedicates nearly all its transistors and power to matrix arithmetic.

Google TPU Evolution

Note the jump from v6e (918 TFLOPS BF16) to v7 (4,614 TFLOPS FP8). Part of this is architectural improvement; part is the shift from BF16 to FP8. The trend is clear: each generation pushes precision lower and throughput higher, because neural network workloads tolerate it.

On-Device NPUs

The same principle—dedicated matrix-multiply silicon—has migrated into phones and laptops. Apple’s Neural Engine, Qualcomm’s Hexagon NPU, and Intel’s NPU are all small systolic arrays integrated into the SoC (system on chip). They handle on-device AI tasks: face recognition, voice processing, image enhancement, and local LLM inference. Their throughput is measured in TOPS (tera-operations per second, usually INT8):

These numbers are modest compared to data-center accelerators, but they enable real-time AI inference without a network round-trip. An 80 TOPS NPU can run a small language model (7–13 billion parameters, quantized to 4-bit) at conversational speed entirely on-device.

Architecture

An ASIC is the logical endpoint of specialization. Where a CPU is fully general, a GPU is parallel-general, and an NPU is matrix-specialized, an ASIC is algorithm-specific. The entire chip is a hardwired implementation of one computation. There is no instruction decoder, no register file, no program counter. Data flows through fixed logic gates that implement the target algorithm directly.

The most visible example is Bitcoin mining. The SHA-256 hash function is a fixed sequence of bitwise operations. An ASIC miner implements this sequence as a physical circuit. The Antminer S21 XP Hydro (2024) computes 473 trillion SHA-256 hashes per second. Each hash involves roughly 12,000 32-bit integer operations, so this is equivalent to approximately $5.7 \times 10^{18}$ integer operations per second—a number that exceeds the FP64 throughput of the world’s fastest supercomputer, for a single rack-mounted device. But it can do nothing else. It cannot add two numbers. It cannot run a program. It computes SHA-256, and that is all.

AI-Specific ASICs: Wafer-Scale

The most ambitious ASIC for AI is the Cerebras Wafer-Scale Engine (WSE). Instead of cutting a silicon wafer into hundreds of individual chips, Cerebras uses the entire wafer as a single processor:

125 petaflops on a single wafer. That is roughly equivalent to 62 NVIDIA H100 GPUs—but in a single device with 44 GB of on-chip SRAM (no off-chip memory bandwidth bottleneck). The WSE-3 uses TSMC 5nm and draws about 15 kW. Its advantage is that the entire model can reside in on-chip memory, eliminating the memory-bandwidth bottleneck that limits GPU training.

Graphcore’s Intelligence Processing Unit (IPU) takes a different approach: 1,472 independent processor tiles, each with 6 hardware threads, and 900 MB of in-processor SRAM. The Bow IPU (2022) achieves 350 TFLOPS at FP16 per chip.

The ASIC Tradeoff

The advantage of an ASIC is raw efficiency: more operations per watt and per dollar than any general-purpose processor. The disadvantage is that if the algorithm changes, the chip is worthless. Bitcoin miners become paperweights if the network switches hash functions. AI ASICs become obsolete if neural network architectures shift in ways the hardware cannot accommodate. This is why GPUs, despite being less efficient per-operation, dominate AI training: they can run any computation, so they are not stranded by algorithmic change.

Architecture

An FPGA is a grid of configurable logic blocks connected by a programmable interconnect. You describe your desired circuit in a hardware description language (Verilog or VHDL), and a synthesis tool maps it onto the FPGA’s physical resources. The result is custom hardware—but hardware that can be erased and reconfigured in milliseconds.

FPGAs occupy a middle ground. They are $3$–$10\times$ slower than an equivalent ASIC (because the programmable routing adds delay and area overhead) but $10$–$100\times$ more power-efficient than a CPU for the same algorithm. They excel in applications that need hardware-speed processing but where the algorithm may change: telecommunications, financial trading (low-latency order execution), video encoding, and prototyping ASICs before committing to fabrication.

Modern FPGAs have integrated hardened blocks for common operations. AMD/Xilinx Versal devices include dedicated AI Engine tiles (systolic arrays for matrix math) alongside the traditional FPGA fabric:

These numbers are modest compared to GPUs or NPUs, but FPGAs achieve them at much lower latency (microseconds, not milliseconds) and with deterministic timing. In high-frequency trading, an FPGA can execute a trading strategy in 1–3 microseconds, while a CPU takes 10–50 microseconds. That difference is worth billions of dollars per year to quantitative trading firms.