# Nvidia won training. Inference is a different problem entirely

> AI training vs inference explained: training rewards raw compute, inference is bottlenecked by memory bandwidth. That is why every new chip leads with memory.

- Published: 2026-09-10
- Topic: [Computing](https://futuretechnologyhq.com/topic/computing/)
- URL: https://futuretechnologyhq.com/article/ai-training-vs-inference-chip-war/
- Publisher: [Future Technology](https://futuretechnologyhq.com/), free to read. Quote as "Future Technology" with a link to the article.

## Key points

- Training is a one off cost per model generation, while inference is paid on every single request
- Training rewards raw compute throughput, but the decode phase of inference is bottlenecked by memory bandwidth and capacity
- AMD Helios ships 72 accelerators with 31 terabytes of memory against 20.7 terabytes in the Nvidia equivalent, and loses on headline performance
- If a hardware launch leads with FLOPS it is a training pitch, and if it leads with terabytes it is aimed at inference

## Article

Training a frontier model is a one off cost. Inference is a bill that arrives every time somebody uses it. That single difference explains why hardware announcements now lead with memory numbers instead of teraflops.

## AI training vs inference explained

Training is the process of building the model. It is enormous, expensive, done once per model generation, and then it is finished. Inference is what happens when a user sends a prompt and the model answers. One is capital expenditure. The other behaves like a utility bill that scales with how popular you are.

| | Training | Inference |
|---|---|---|
| When it happens | Once per model generation | Every request |
| Who pays | The lab building the model | Whoever serves the model, and the user |
| Dominant hardware trait | Raw compute throughput, FLOPS | Memory bandwidth and capacity |
| Typical duration | Weeks to months, continuous | Milliseconds to seconds, constant |
| Cost shape | Large, predictable, front-loaded | Small per unit, unbounded in aggregate |

## Why inference needs memory bandwidth

Inference splits into two phases. Prefill reads the prompt and is compute-heavy, so it behaves a little like training. Decode produces the answer one token at a time, and for every token the accelerator has to read the model weights and the accumulated context out of memory again.

That read is the bottleneck. The arithmetic per token is trivial by modern standards; moving the weights is not. Add a long context window and the key-value cache grows until memory capacity, rather than compute, decides how many users one machine can serve.

In plain terms: a chip that is twice as fast at maths but has the same memory bandwidth will not serve twice as many tokens.

## The hardware follows the constraint

AMD Helios ships 72 accelerators with 31 terabytes of memory against 20.7 terabytes in the Nvidia equivalent, and quietly loses on headline performance. That is not a compromise, it is the design goal.

Qualcomm's [AWS agreement](https://nand-research.com/qualcomm-aws-ink-multi-generation-deal-for-custom-ai-silicon-optical-connectivity/) is aimed explicitly at decode-heavy inference, with a memory capacity optimised design it calls High Bandwidth Compute. The [commercial terms of that deal](https://thenextweb.com/news/qualcomm-amazon-4bn-warrants) tell you how badly Qualcomm wanted the reference customer. TSMC is mass producing co-packaged optics to cut the energy cost of moving data between chips, which is the same problem viewed from the interconnect side.

Nvidia still makes the best training hardware there is. The market underneath it moved.

## Why your own bills work the same way

The physics does not change at small scale. If you run models locally, the ceiling is usually how much memory you have and how fast it is, not how many cores. That is why home setups tend to be memory-first builds, and why the [cost of the drives](/article/nas-drives-cost-more-than-the-box-2026) and modules around the box now matters more than the box.

It is also why [national compute programmes](/article/china-532-billion-ai-compute-plan-2030) are measured in memory supply as much as in accelerator counts.

## The part worth watching

Watch which number a vendor leads with. If a launch opens with FLOPS, it is a training pitch. If it opens with terabytes and bandwidth, it is aimed at the market that pays every day. Most of the [frontier models released this month](/article/ai-model-releases-september-2026) are being served far more often than they were ever trained.

For a local setup, memory capacity is the practical limit long before compute is. A 32GB DDR5 kit such as the [Corsair Vengeance 6000MHz CL30](https://www.amazon.co.uk/dp/B0BPTKD797?tag=futuretech0d6-21) is the sort of upgrade that changes which models will load at all.

Some links in this article are affiliate links. We may earn a small commission at no extra cost to you.

## Key takeaways

- Training a frontier model is a one-off cost, while inference is a recurring expense that scales with usage.
- The difference between training and inference explains why hardware announcements now focus on memory numbers rather than teraflops.
- Inference is a utility-like bill that behaves similarly to a utility bill, scaling with how popular a model is.
- Inference is bottlenecked by memory bandwidth, with the read operation being the main limitation.
- The hardware industry has shifted to prioritise memory capacity and bandwidth over raw compute throughput for inference workloads.
- Vendors now lead with memory numbers and bandwidth in their announcements, indicating a shift towards serving inference workloads.

## Sources

- [NAND Research: Qualcomm and AWS custom AI silicon and optical connectivity](https://nand-research.com/qualcomm-aws-ink-multi-generation-deal-for-custom-ai-silicon-optical-connectivity/)
- [The Next Web: Qualcomm and Amazon warrants deal](https://thenextweb.com/news/qualcomm-amazon-4bn-warrants)