When people talk about AI chips, they tend to focus on the compute numbers of the GPU, such as how many TFLOPS or how much FP4 performance it has. But what really determines whether an AI chip can actually run, and how fast it runs, is often not the compute itself, but the ring of high-bandwidth memory called HBM sitting next to it.
HBM is no longer “memory” in the traditional sense. It is more like part of the GPU, and it is one of the most strained, hardest-to-make, and most concentrated links in the entire AI compute system.
To understand HBM’s place, you first have to understand one concept: the memory wall.
GPU compute has increased by dozens of times over the past decade and more, but the bandwidth of ordinary DRAM has grown much more slowly. Large model training and inference are both classic bandwidth-sensitive workloads. The weight parameters alone can easily reach hundreds of GB, and every step of computation has to move massive amounts of data from memory. If bandwidth cannot keep up, compute sits idle, and the GPU becomes an expensive ornament. That is the so-called memory wall.
HBM solves exactly this problem.
Instead of placing memory on a PCB next to the GPU, it uses through-silicon vias (TSV) to vertically stack multiple layers of DRAM dies, then attaches them to the same substrate as the GPU through advanced packaging. With short distances and wide channels, bandwidth can reach more than ten times that of ordinary DDR5.
By NVIDIA’s Blackwell generation, a single B200 already has 192GB of HBM and bandwidth of 8TB/s, nearly two and a half times the 3.35TB/s of the previous-generation H100.
Without HBM, today’s large models simply would not run; without next-generation HBM4, future inference clusters will not be able to handle even larger models.
But what makes HBM difficult is manufacturing.
It is not a stacked version of ordinary DRAM, but a complete set of advanced packaging processes. First, each DRAM layer itself has to achieve a high yield. Then tens of thousands of TSV vias have to be made on the wafer, and 8-layer, 12-layer, or even 16-layer chips have to be bonded together with precision. If any layer has a problem, the entire HBM stack is scrapped.
Yield is multiplied layer by layer, and the more layers there are, the faster yield drops. Industry estimates say that the mass-production yield of 12-layer and 16-layer HBM3E/HBM4 has long struggled around 60%, which means a substantial portion of capacity is inherently waste.
If yield cannot go up, output cannot go up, and prices cannot come down. A single HBM3E stack costs about $300 on the market, while HBM4 has already risen to $550 per stack, and a B200 needs several stacks attached.
Supply is even tighter on the supply side.
Only three companies in the world can make HBM: SK hynix, Samsung, and Micron. SK hynix has long held the key supply position for NVIDIA, Samsung has struggled to catch up in HBM3E, and Micron has actually overtaken Samsung in some share. By the second quarter of 2026, Samsung had 38% of overall DRAM share, SK hynix 25%, and Micron 24%, but in the share allocation for HBM4, SK hynix still dominates.
Three vendors, concentrated capacity, the yield bottleneck of vertical stacking, and the fact that almost all AI chips from NVIDIA, AMD, Google TPU, and Amazon Trainium rely on HBM together make it the most typical scarce resource in the entire AI industry chain.
So HBM’s position today is very clear: it is not memory, but a key component in AI compute systems, the layer in the entire semiconductor stack of “storage, compute, interconnect, packaging, power” that currently has the sharpest bottleneck, the hardest capacity expansion, and the highest concentration.
If you look at AI infrastructure without looking at HBM capacity and yield, you are only seeing half the picture.