Open benchmark v0.2.0
How good are AI models at real Salesforce work?
Not trivia. Apex, Flow and LWC that have to pass hidden tests, and SOQL that has to return the right rows.
Forcebench gives models real Salesforce engineering tasks, runs what they build (a check-only deploy with hidden tests in a clean scratch org, or Jest) and reports how often it works, with honest error bars. Open source and versioned, with every final answer on the public tasks published.
$forcebench leaderboard --api --top 3
$forcebench leaderboard --local --top 3
By lab
The best of each lab
Each lab’s highest-scoring model at its best configuration, ranked on each board separately: API, models reached through a hosted service, and Local, models served on Forcebench’s own hardware. The line on each bar is its 95% interval.
- 95% interval
Show the numbers
| Model | Score | 95% CI |
|---|---|---|
| Claude Opus 5.5 low | 96% | 93–98 |
| Gemini 4 Argon medium | 96% | 93–98 |
| GPT-6 Astra medium | 95% | 92–97 |
| Grok 4.7 high | 92% | 89–95 |
| Muse Wonder low | 87% | 84–91 |
| DeepSeek V4.1 Flash high | 81% | 77–85 |
| MiMo V2.6 Pro thinking on | 71% | 66–75 |
| GLM-5.3 Flash high | 65% | 60–69 |
| Inkling Small high | 60% | 55–66 |
| Mistral Large 4 high | 60% | 55–65 |
| Nemotron 3 Ultra thinking on | 59% | 54–64 |
| Qwen3.8 Flash-Next thinking on | 44% | 39–49 |
| Laguna S 2.1 thinking on | 37% | 32–42 |
| Ling 3.0 Flash thinking on | 36% | 32–41 |
- 95% interval
Show the numbers
| Model | Score | 95% CI |
|---|---|---|
| DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts | 76% | 71–80 |
| GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense | 64% | 59–68 |
| Qwen3.8 Flash-Next medium · NVFP4 | 58% | 53–62 |
| MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts | 51% | 46–56 |
| Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt) | 47% | 42–53 |
| Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt) | 45% | 40–50 |
| Gemma 4 31B thinking on · QAT W4A16 | 43% | 38–48 |
| MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) | 42% | 37–47 |
| Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) | 33% | 28–38 |
| MiniCPM-V 4.7 35B-A3B thinking on · BF16 | 31% | 26–36 |
Leaderboard
Top models, API and Local
The score within three attempts (c@3), each retry shown what the deploy and tests reported, each model at its highest-scoring configuration (as the leaderboard opens), on two boards ranked separately: API, models reached through a hosted service, and Local, models served on Forcebench’s own hardware. Where two bars overlap, those models are not clearly different; where they don't, one is clearly above the other. A configuration still waiting for its retries isn’t ranked yet; it’s added once they finish.
API
| # | Model and best config | Within 3 attempts c@3 · 95% CI |
|---|---|---|
| 1 | Claude Opus 5.5effort low | 96%95% CI 93–98 |
| 1 | Gemini 4 Argoneffort medium | 96%95% CI 93–98 |
| 3 | GPT-6 Astraeffort medium | 95%95% CI 92–97 |
| 4 | GPT-6.1 Soleffort high | 93%95% CI 90–95 |
| 4 | Claude Sonnet 5.5effort high | 93%95% CI 90–96 |
Local
| # | Model and best config | Within 3 attempts c@3 · 95% CI |
|---|---|---|
| 1 | DeepSeek V4.1 Flash 1017Native FP8, FP4 experts · effort high | 76%95% CI 71–80 |
| 2 | DeepSeek V4.1 FlashEXL3 2.9bpw · effort high | 73%95% CI 68–78 |
| 3 | GLM-5.3 FlashEXL3 4.0bpw experts, FP8 dense · effort high | 64%95% CI 59–68 |
| 4 | Qwen3.8 Flash-NextNVFP4 · effort medium | 58%95% CI 53–62 |
| 5 | DeepSeek V4 Flash Vision experimentalFP8 + FP4 experts · effort high | 52%95% CI 47–57 |
How the best local configuration compares with the API ones, score, tokens and time: the local vs API study.
Findings
What the numbers say
Worked out from the latest results every time the site builds, not written by hand. Each comes from one board and compares configurations on it only, except local vs API. Each names its metric, the same as the chart or table its link opens: c@3 for rankings, pass@1 for comparisons paired task by task on first answers.
Claude Opus 5.5 leads at 96% within three attempts
But 5 other API models, each at its best configuration, overlap its 95% interval of 93–98%, so it is not clearly different from them. Nemotron 3 Super isn’t ranked yet: its retries haven’t finished.
DeepSeek V4.1 Flash 1017 leads at 76% within three attempts
But 1 other local model, at its best configuration, overlaps its 95% interval of 71–80%, so it is not clearly different from that one.
Best local: DeepSeek V4.1 Flash 1017 at 76% within three attempts
12 of 29 API models, each at its best configuration, are clearly above it. The best API model, Claude Opus 5.5 (effort low, reasoning on), scores 96%: clearly above it.
EXL3 4.0bpw experts, 4-bit dense vs EXL3 4.0bpw experts, FP8 dense: −6 pts pass@1
GLM-5.3 Flash at EXL3 4.0bpw experts, 4-bit dense, against its EXL3 4.0bpw experts, FP8 dense reference run, pass@1 on the same tasks: clearly lower (−6 pts, 95% CI −12 pts to −0.7 pts). The engine changed too (vLLM + ExLlamaV3 to TensorFold), so not all of it can be put down to quantisation.
More thinking: clearly higher for Claude Sonnet 5
Going from effort low to high took Claude Sonnet 5 from 55% to 65% pass@1, compared on the same tasks: clearly higher (+10 pts, 95% CI +5 pts to +14 pts). Mean output tokens went from 730 to 2.4k.
Salesforce docs is the hardest suite
Of the API models, each at its highest-scoring configuration, the best there is Claude Opus 5.5 (effort low), at only 82% within three attempts.
Sovereign AI
Models you own, tested on your work
Open-weight models are models a team can own, so the data stays on infrastructure you control. If you are choosing one, Azul Labs can benchmark your shortlist on your kind of Salesforce work for a fixed fee and report back.
How it works
Graded the way a deploy would grade it
- 1
Real tasks
Fifteen suites of the work Salesforce developers actually do, including Apex, Lightning Web Components, Permissions & access, Packaging and CI/CD. Not multiple-choice trivia.
- 2
Run, not judged
Answers are deployed as a check-only validation to a clean scratch org with hidden tests, run under Jest, or checked by deterministic validators. No model grades another model's work.
- 3
Honest numbers
Ranked within three attempts (c@3): after a failed answer the model sees what the deploy, tests or tools reported and tries again, the way a developer works, with 95% confidence intervals and the exact quantisation, engine and effort of every configuration. Final answers, grades and token counts for public tasks are published in the repo; reasoning traces are available on request through the gated Hugging Face dataset; nothing from the private pool is published except approved aggregate scores. Read the methodology.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au