What Is HBM4, and Why Your AI Accelerator Is Starving Rather Than Thinking

AMD is quoting 31TB of memory per rack while admitting a lower headline inference FLOPS number than Nvidia. That is not a slip in the marketing. It is the entire argument.

In the Helios versus Vera Rubin comparison, AMD puts 31TB of HBM4 against Nvidia 20.7TB, roughly 50 percent more, in a rack that loses on peak compute. High bandwidth memory is why that trade can still be the right one.

What HBM actually is

Ordinary DDR memory sits on sticks a few centimetres from the processor and talks over a comparatively narrow bus. HBM stacks DRAM dies vertically, connects them with vertical channels through the silicon, and mounts the whole stack on the same package as the compute die. Shorter distance, far wider bus, much more data moved per second per watt.

It is expensive to make and awkward to cool, which is why it shows up in accelerators and not in laptops. The longer version, including how it compares against GDDR, is in our HBM versus GDDR versus DDR explainer.

Capacity and bandwidth are not the same number

Capacity is how much model you can hold. Bandwidth is how fast you can move it. People confuse the two constantly, and spec sheets encourage it by putting both in the same table.

The distinction decides real performance. When a model generates a token, the accelerator has to read the relevant weights out of memory. Do that again for the next token, and the next. If reading takes longer than the arithmetic, the compute units sit idle waiting. That is a memory bound workload, and most inference is memory bound. Adding FLOPS to a memory bound system buys you nothing.

This is the same constraint that decides what runs on a desktop. If you have ever worked out how much VRAM a local model needs, you have already done the rack scale calculation in miniature.

Three companies, one bottleneck

HBM4 is the current generation, succeeding HBM3E, and it is produced at volume by exactly three firms: SK hynix, Samsung and Micron. Nobody else is close. That concentration is why HBM supply, rather than accelerator design, has set the pace of the last two years, and why memory contracts get signed years ahead of the chips that use them.

It also shapes the competitive picture further down the stack. AMD gains in x86 server share rest partly on being able to buy the memory to fill its racks.

For anyone building a machine to run models locally rather than renting rack time, system memory is the same problem at a smaller scale, and a 32GB DDR5 kit such as the Corsair Vengeance RGB DDR5 6000MHz CL30 is the usual floor before VRAM becomes the limit.

Next time a chip launch leads with a compute number, find the memory line before you decide what it means. It is usually the one that tells you what the machine will actually do.

Disclosure: some links above are Amazon Associates links. If you buy through them Future Technology earns a small commission at no extra cost to you.