Test local LLM speed before you buy the hardware
Key takeaways
- A Show HN project called Llmspeed reached 323 points and 86 comments by letting you test a local model's speed before you buy hardware.
- Memory comes first: a model has to fit in VRAM or system RAM, and quantisation is what shrinks it to fit.
A Show HN project called Llmspeed attracted 323 points and 86 comments 2 days ago by answering one question: how fast will a local model actually run on the hardware you are thinking of buying?
The answer has two parts: whether the model fits, and how fast it runs once it does.
How much RAM do you need to run a local LLM
Start with fit.
Quantisation is the technique that makes this work. It stores the model's weights at lower precision, which shrinks the file at a modest cost in quality. A model that would never fit at full precision may fit comfortably once quantised.
If a model does not fit, the system spills into slower system memory and speed collapses. Check fit before anything else.
What tokens per second feels like
Speed is measured in tokens per second. For chat, a rate close to comfortable reading speed is enough. Coding assistants, which produce long outputs and often run several steps, feel better when they are quicker.
A reviewer tested NVIDIA's Nemotron 3.5 Bolt locally on a Surface Laptop Ultra and achieved up to 65.5 tokens per second. That is a laptop, not a tower with a flagship card, which matters when you are deciding how much to spend.
The Nemotron 3.5 Bolt 74B-A9B model has 74 billion total parameters with roughly 9 billion active per token, using a mixture-of-experts architecture. It relies on native 2-bit quantisation to keep the 24GB footprint manageable while preserving capability.
A buying checklist
Pick the model you want to run first, then work backwards. Work out its quantised size, add headroom for context window, and match that to your available memory. Only then compare speed.
What to watch
Tools like Llmspeed are worth following. If you can test the feel of a model before spending, you avoid both overbuying and the more expensive mistake of buying a machine that is too slow to use.
Some links in this article are affiliate links. We may earn a small commission at no extra cost to you.
