Models
Every model, every configuration
33 models. Each page lists every configuration of the model (effort, quantisation, engine, service) on the API board, models reached through a hosted service, and the Local board, models served on Forcebench’s own hardware.
| Model | Developer | Weights | Board | Configurations | Highest on API within 3 attempts (c@3) | Highest on Local within 3 attempts (c@3) |
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | Anthropic | Closed | API | 1 | 88% | — |
| Claude Haiku 4.5 | Anthropic | Closed | API | 2 | 50% | — |
| Claude Haiku 5.5 | Anthropic | Closed | API | 4 | 76% | — |
| Claude Opus 4.8 | Anthropic | Closed | API | 4 | 88% | — |
| Claude Opus 5.5 | Anthropic | Closed | API | 3 | 96% | — |
| Claude Sonnet 5 | Anthropic | Closed | API | 4 | 77% | — |
| Claude Sonnet 5.5 | Anthropic | Closed | API | 6 | 93% | — |
| DeepSeek V4 Flash Vision experimental | DeepSeek | Open | Local | 1 | — | 52% |
| DeepSeek V4 Pro | DeepSeek | Open | API | 1 | 69% | — |
| DeepSeek V4.1 Flash | DeepSeek | Open | API and Local | 5 | 81% | 73% |
| Gemini 3.8 Flash | Closed | API | 1 | 89% | — | |
| Gemma 4 26B-A4B | Open | Local | 2 | — | 40% | |
| Gemma 4 31B | Open | Local | 2 | — | 43% | |
| GLM-5.3 | Z.ai | Open | Local | 2 | — | lite subset only |
| GLM-5.3 Flash | Z.ai | Open | API and Local | 11 | 65% | 64% |
| GPT-5.6 Sol | OpenAI | Closed | API | 1 | 88% | — |
| GPT-6 Astra | OpenAI | Closed | API | 2 | 95% | — |
| GPT-6 Luna | OpenAI | Closed | API | 1 | 82% | — |
| GPT-6.1 Sol | OpenAI | Closed | API | 1 | 93% | — |
| Grok 4.6 | xAI | Closed | API | 1 | 89% | — |
| Grok 4.7 | xAI | Closed | API | 1 | 92% | — |
| Inkling Small | Thinking Machines Lab | Open | API | 1 | 60% | — |
| Laguna S 2.1 | Poolside | Open | API | 1 | 37% | — |
| Ling 3.0 Flash | inclusionAI | Open | API | 1 | 36% | — |
| MiMo V2.6 Flash | Xiaomi | Open | API and Local | 3 | 55% | 51% |
| MiMo V2.6 Pro | Xiaomi | Open | API | 1 | 71% | — |
| Mistral Large 4 | Mistral AI | Closed | API | 1 | 60% | — |
| Nemotron 3 Super | NVIDIA | Open | API | 1 | retries pending | — |
| Nemotron 3 Ultra | NVIDIA | Open | API | 1 | 59% | — |
| Nemotron 3.5 Lightning | NVIDIA | Open | API | 1 | 20% | — |
| Qwen3.6 35B-A3B | Qwen | Open | Local | 2 | — | 34% |
| Qwen3.8 27B | Qwen | Open | Local | 6 | — | 37% |
| Qwen3.8 Flash-Next | Qwen | Open | API and Local | 5 | 44% | 58% |
- retries pending Every task has its first answer graded, but its retries haven’t finished: each failed answer gets up to two more attempts, shown what the deploy and tests reported. The board ranks within three attempts, so it has no score there until they have.
- lite subset only Run only on the lite subset (60 tasks, for effort sweeps), which is never ranked with the full benchmark. Its page has the results.
- — No configuration on that board.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au