Nvidia won training. Inference is a different problem entirely

Training a frontier model is a one off cost. Inference is a bill that arrives every time somebody uses it. That single difference explains why hardware announcements now lead with memory numbers instead of teraflops.

AI training vs inference explained

Training is the process of building the model. It is enormous, expensive, done once per model generation, and then it is finished. Inference is what happens when a user sends a prompt and the model answers. One is capital expenditure. The other behaves like a utility bill that scales with how popular you are.

| | Training | Inference | |---|---|---| | When it happens | Once per model generation | Every request | | Who pays | The lab building the model | Whoever serves the model, and the user | | Dominant hardware trait | Raw compute throughput, FLOPS | Memory bandwidth and capacity | | Typical duration | Weeks to months, continuous | Milliseconds to seconds, constant | | Cost shape | Large, predictable, front-loaded | Small per unit, unbounded in aggregate |

Why inference needs memory bandwidth

Inference splits into two phases. Prefill reads the prompt and is compute-heavy, so it behaves a little like training. Decode produces the answer one token at a time, and for every token the accelerator has to read the model weights and the accumulated context out of memory again.

That read is the bottleneck. The arithmetic per token is trivial by modern standards; moving the weights is not. Add a long context window and the key-value cache grows until memory capacity, rather than compute, decides how many users one machine can serve.

In plain terms: a chip that is twice as fast at maths but has the same memory bandwidth will not serve twice as many tokens.

The hardware follows the constraint

AMD Helios ships 72 accelerators with 31 terabytes of memory against 20.7 terabytes in the Nvidia equivalent, and quietly loses on headline performance. That is not a compromise, it is the design goal.

Qualcomm's AWS agreement is aimed explicitly at decode-heavy inference, with a memory capacity optimised design it calls High Bandwidth Compute. The commercial terms of that deal tell you how badly Qualcomm wanted the reference customer. TSMC is mass producing co-packaged optics to cut the energy cost of moving data between chips, which is the same problem viewed from the interconnect side.

Nvidia still makes the best training hardware there is. The market underneath it moved.

Why your own bills work the same way

The physics does not change at small scale. If you run models locally, the ceiling is usually how much memory you have and how fast it is, not how many cores. That is why home setups tend to be memory-first builds, and why the cost of the drives and modules around the box now matters more than the box.

It is also why national compute programmes are measured in memory supply as much as in accelerator counts.

The part worth watching

Watch which number a vendor leads with. If a launch opens with FLOPS, it is a training pitch. If it opens with terabytes and bandwidth, it is aimed at the market that pays every day. Most of the frontier models released this month are being served far more often than they were ever trained.

For a local setup, memory capacity is the practical limit long before compute is. A 32GB DDR5 kit such as the Corsair Vengeance 6000MHz CL30 is the sort of upgrade that changes which models will load at all.

Some links in this article are affiliate links. We may earn a small commission at no extra cost to you.