Where the best local models land

How the tokens were counted

Output tokens include reasoning everywhere on this page. They come from the usage each provider or server reports (what it bills, for a vendor's API), never from counting the text that comes back, which for some models is only a summary of their reasoning, or none of it:

• Local inference servers count every token they generate, thinking included.
• Anthropic's API counts thinking as output, including thinking it returns only as a summary or not at all.
• Some services leave reasoning out of the output count and report it only in the request's total. Forcebench adds it back.

Each model family uses its own tokeniser, so the same text can be a different number of tokens across vendors. Treat token comparisons across vendors as approximate.

The mean, the median and the 90th percentile are over each configuration's answers. A few very long answers pull a mean well above the median, which is why both are shown.

Tokens are the fairer measure of cost. Time per answer mostly measures the hardware and how many requests ran at once: the local models here ran on hardware a small team could own, one to eight requests at a time depending on the run, while a vendor's API runs on data-centre hardware.

What to keep in mind

• Local models run quantised to fit the hardware (the best one at EXL3 2.9bpw). The quantisation study shows what changes with it. API providers don't say how they serve their models.

The best local model is not clearly different from some API models running with thinking off, and clearly below the best API ones. The reasons to run your own are data control and a model that doesn't change under you. See Sovereign AI.