57 tokens per second instead of 30 on the same Mac. How does Magnitude beat llama.cpp?

30 and 57. That is how many tokens per second the same model produces on the same Mac M4 Pro: the first number in llama.cpp, the second in Magnitude. The hardware stayed the same; what changed were the kernels that run the model. For an agent that generates tens of thousands of tokens per session, that means waiting almost half as long at every step.

Cloud tokens for an agent cost real money: we worked out what a task actually costs with GPT-6 Sol and Opus 5.5. So an engine that squeezes extra tokens out of a laptop matters to anyone who has thought about moving part of an agent's work to a local model.

What Magnitude is and why it beats llama.cpp

Magnitude is built by Anders and Tom, a YC S25 team. Their earlier product was a browser agent, which lives in a neighbouring repository and has 4,134 stars. The magnitudedev/magnitude repository itself went through three lives over the summer: in June it was a CLI with a cloud key and credits, in August an agent running on local models, and since mid-September an engine plus a desktop app. So some of its 5.9k stars as of 1 October were earned by earlier versions of the project.

The idea is simple. llama.cpp and Ollama ship kernels compiled ahead of time for a wide class of hardware. Chip-specific engines are faster, the authors say, but support less. Magnitude writes kernels with tunable parameters and picks the values on your device. The engine is written in Rust, with its own GPU kernel runtime and autotuner. It runs on Apple Silicon, NVIDIA, AMD (via Vulkan) and plain CPU, on macOS, Linux and Windows.

Kernels that tune themselves to your chip in a minute

Tuning happens once: when you download a new model, the engine spends about a minute trying kernel parameters until the gains stop growing. After that the model runs on the configuration it found.

The other half of the speed comes from memory. Magnitude stores the KV cache, where the model keeps the context it has read, with 8 bits for keys and 4 bits for values. According to the authors, the cache takes half the space and decode gets faster as a result. The rest is designed for agents rather than chat:

• context memory is not reserved upfront; it grows with the session and is released when the agent stops;
• parallel sessions that start the same way (system prompt, tool schemas) share a common prefix cache;
• a model loads when an agent asks for it and unloads after it sits idle.

The catalogue is small for now: 15 models from six families, from Qwen and Gemma to Nemotron and Liquid LFM, ranging from 1.1B to 35B. Quantization is 4 bits and up only: below that, the authors say, models' reasoning and tool calls fall apart.

How to connect Claude Code, Codex or Cline to a local model

Setup goes through the app: download it from magnitude.dev, pick a model in the Discover tab, connect an agent in Connections. The magnitude CLI ships with the app. Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi and Cline connect in one click; everything else goes through an OpenAI-compatible endpoint.

With Claude Code there is one detail worth knowing in advance. The Connect button rewrites your ~/.claude/settings.json and routes Claude Code through the Magnitude gateway:

"ANTHROPIC_BASE_URL": "http://ADDRESS:10100/inference/anthropic/proxies/claude-code",
"CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY": "1",
"model": "anthropic-local/MODEL_ID"
claude --model anthropic-local/MODEL_ID

Requests to cloud models also pass through the gateway, so Magnitude has to stay running for as long as Claude Code is connected. To go back to normal, click Disconnect and restart Claude Code. If you already have your own variables in settings.json, back the file up before connecting.

Where "twice as fast as llama.cpp" comes from

Magnitude's headline number was measured on one model and two machines:

The benchmark task is synthetic: the prompt is filled with Moby-Dick up to 64 thousand tokens and the model is asked to repeat the last passage. On HN the authors were open about how they configured llama.cpp: flash attention on, 16-bit KV cache. They tried quantizing its cache to 8 and 4 bits, as in their own engine, but llama.cpp's decode dropped when they did. So in this comparison Magnitude's cache is almost half as heavy, and a lighter cache, by the authors' own account, speeds up decode. Part of the gain comes from that, not only from kernel tuning.

There is a second subtlety, and we found it in their own methodology. In session-bench, Magnitude runs the MLX version of the model and llama.cpp runs the GGUF, and the document says plainly that a shared model name does not prove the files are equivalent. The number is honest, but it describes the combination of engine, format and cache, not the kernels in isolation. The authors have not yet published a comparison with MLX; they say it is coming soon.

Where Magnitude is still slower

In the first day on HN, users ran it on their own hardware. The most visible gap is on the newest Macs:

The authors named the M5 cause themselves: the kernels do not yet use the Metal 4 matrix operations these chips introduced. According to one user, llama.cpp closed the same prefill gap only recently, in build b10853. Magnitude does not support several GPUs in one machine yet; that is on the near-term roadmap. Two more bugs surfaced on day one: a long model evaluation step before the first download, and understated recommendations on machines with a single 16 GB card.

What we take from it

Our opinion, and you are free to disagree: Magnitude is worth trying if you run agents on a Mac with an M4 Pro or M4 Max and decode speed on long context is your bottleneck. The gain there is claimed on an open methodology, and you can reproduce it on your machine with session-bench. We will be wrong if the promised MLX comparison shows that a regular MLX engine on a Mac is no slower: then Magnitude stays a convenient wrapper rather than a new class of engine.

On M5, on consumer NVIDIA cards and on multi-GPU setups it is better to wait for the kernel updates. More interesting than the speed is the idea itself: an engine built for long agent sessions, with a shared cache and memory that grows and then lets go. We looked at what the harness around a model costs when we compared Claude Code, Codex and pi on price, and a local engine covers exactly the part of the bill that goes to tokens.

Versions and figures as of 1 October 2026; check the repository for current ones.