Samsung's New Memory Chips Stack Straight on Top of AI Accelerators

At the Future of Memory and Storage conference this month, Samsung showed three memory technologies that treat the old rulebook as optional. zHBM, zNAND-O and BV-NAND all lean on vertical wafer bonding, stacking memory dies directly on top of the processor rather than wiring them in next to it.

What zHBM actually does

High Bandwidth Memory already stacks DRAM dies on top of each other and sits them beside a GPU or AI accelerator, connected by a silicon interposer. zHBM goes a step further: it bonds the memory stack directly onto the accelerator package using micro-bump bonding, cutting the distance data has to travel almost to zero.

Samsung's claims are steep. It says zHBM can hit roughly 8 times the speed of current HBM5, with about 10 times the density in the same footprint, a third of the energy use, and half the thermal resistance. Those numbers come from Samsung, not independent testing, and the technology is still at an early concept stage with no confirmed production timeline. Take the multipliers as a direction of travel rather than a spec sheet.

zNAND-O and BV-NAND apply similar bonding ideas to flash storage, aimed at pushing density up without the usual jump in power draw that comes from adding more layers to a NAND stack.

Why it matters

AI accelerators are increasingly bottlenecked by how fast they can move data in and out of memory, not by raw compute. Nvidia, AMD and everyone building next generation AI silicon are chasing the same problem from different angles, and memory bandwidth per watt is now as competitive a battleground as core counts once were.

Stacking memory directly onto the chip package, rather than routing it through an interposer, shortens the electrical path and cuts the energy lost moving data back and forth. If Samsung's numbers hold up anywhere close to what it is claiming, it changes the maths for how much a data centre operator pays to run a given amount of AI inference.

The catch is that early concept stage covers a lot of ground. Getting a bonding process from a conference demo to something SK Hynix, Micron or TSMC's packaging lines can match, and that Nvidia is willing to design a Vera Rubin generation product around, usually takes years, not months. Samsung has form here, it talked up HBM4 concepts well ahead of actual production ramp.

Still, this is where the real AI hardware race is happening now, quietly, away from the GPU keynotes. Whoever solves the memory bandwidth problem first gets to sell every AI lab on the planet a faster, cheaper way to run the models they have already built. Samsung wants to be the company holding that solution.

Some links in this newsletter may be affiliate links. We only recommend products we genuinely think are worth your time.