Leaderboard

Which models can actually do Salesforce work?

Two boards, ranked separately: API, models reached through a hosted service, and Local, models served on Forcebench’s own hardware. This is the Local board. It opens on each model at its best configuration; All configurations has one row per configuration: a model, its quantisation, the engine that served it, the reasoning effort it ran at and the hardware it ran on. Scores are the share of tasks solved, with the uncertainty drawn in.

Benchmark v0.2.0272 tasks in 15 suites9 models21 local configurations14 lite effort-sweep runs not ranked hereUpdated 12 October 2026Download JSON
Filters
Reasoning
Models

API: models reached through a hosted service. Local: models served on Forcebench’s own hardware.

Score by model

Each bar is the share of tasks solved within three attempts (c@3), with its 95% interval: after a failed answer, the model was shown what the deploy, tests or tools reported. Pick pass@1, the first answer alone, or one suite under Score. How it's scored

Within 3 attempts (c@3), by model
Bars whose intervals overlap are not clearly different: point at or tap one to highlight which. Ranks marked = are ties: the same score to the whole percent.
  • 95% interval
1DeepSeek V4.1 Flash high · EXL3 2.9bpwbest of 2 configurations
73%, 95% interval 68–78%
2GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 densebest of 4 configurations
64%, 95% interval 59–68%
3Qwen3.8 Flash-Next medium · NVFP4best of 4 configurations
58%, 95% interval 53–62%
4DeepSeek V4 Flash Vision experimental high · FP8 + FP4 experts1 configuration
52%, 95% interval 47–57%
5MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts1 configuration
51%, 95% interval 46–56%
6Gemma 4 31B thinking on · QAT W4A161 configuration
43%, 95% interval 38–48%
7Gemma 4 26B-A4B thinking on · NVFP41 configuration
40%, 95% interval 35–44%
8Qwen3.8 27B low · MLX 8-bitbest of 6 configurations
37%, 95% interval 33–42%
9Qwen3.6 35B-A3B thinking on · NVFP41 configuration
34%, 95% interval 29–39%
    Show the numbers
    ModelScore95% CI
    DeepSeek V4.1 Flash high · EXL3 2.9bpw · best of 2 configurations73%68–78
    GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense · best of 4 configurations64%59–68
    Qwen3.8 Flash-Next medium · NVFP4 · best of 4 configurations58%53–62
    DeepSeek V4 Flash Vision experimental high · FP8 + FP4 experts · 1 configuration52%47–57
    MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts · 1 configuration51%46–56
    Gemma 4 31B thinking on · QAT W4A16 · 1 configuration43%38–48
    Gemma 4 26B-A4B thinking on · NVFP4 · 1 configuration40%35–44
    Qwen3.8 27B low · MLX 8-bit · best of 6 configurations37%33–42
    Qwen3.6 35B-A3B thinking on · NVFP4 · 1 configuration34%29–39

    Breakdown by suite

    Each suite's score on the page's metric, the same rows in the same order as the chart above: within three attempts (c@3) while the page is on c@3, the first answer (pass@1) when you pick pass@1 under Score. Pick a suite's name to rank the page by it.

    c@3 by suite
    Darker is higher on light backgrounds, brighter is higher on dark ones; every cell prints its score.
    c@3 on each suite, in percent, for each model at its best configuration, in the order of the chart above. A dash: no complete score on that suite.
    Model
    DeepSeek V4.1 Flashhigh · EXL3 2.9bpw957882852450618989834480449594
    GLM-5.3 Flashhigh · EXL3 4.0bpw experts, FP8 dense905682851233338994833967399556
    Qwen3.8 Flash-Nextmedium · NVFP480447155241728681006144601710094
    DeepSeek V4 Flash Vision experimentalhigh · FP8 + FP4 experts605053751217178494444460399044
    MiMo V2.6 Flashthinking on · FP8 + MXFP4 experts8539354512441774785639731110056
    Gemma 4 31Bthinking on · QAT W4A167050591501711898961174767539
    Gemma 4 26B-A4Bthinking on · NVFP47039351506178983611140119028
    Qwen3.8 27Blow · MLX 8-bit704429300017633956226068044
    Qwen3.6 35B-A3Bthinking on · NVFP445392915017116844502840177039

    Every number

    How to read this. Best per model shows each model's highest-scoring configuration. Models run in more configurations get more chances at a high score, so check the N badge. Local configurations only, ranked among themselves. Scores are within three attempts (c@3): after a failed answer, the model is shown what the deploy, tests or tools reported and answers again. Each has a 95% confidence interval. A configuration still waiting for its retries is listed but not ranked yet, and pass@1, the first answer alone, is a choice away under Score. The dot is the score and the bar the interval. Where two configurations’ bars overlap, they are not clearly different; where they don’t, one is clearly above the other. Latency depends on the hardware, so compare it only between configurations on the same hardware. Methodology

    Forcebench leaderboard: Local board. Column headers are buttons that sort the table.
    1DeepSeek V4.1 FlashEXL3 2.9bpw · vLLM + ExLlamaV3 · high · 2× NVIDIA DGX Spark (GB10, 128 GB each)EXL3 2.9bpwvLLM + ExLlamaV3high
    73%95% CI 68–78
    		9578828524506189898344804495947.4k3.5k · 22k161 s
    2GLM-5.3 FlashEXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · highEXL3 4.0bpw experts, FP8 densevLLM + ExLlamaV3high
    64%95% CI 59–68
    		9056828512333389948339673995563.4k1.1k · 10k86 s
    3Qwen3.8 Flash-NextNVFP4 · vLLM · medium · 2× NVIDIA DGX Spark (GB10, 128 GB each)NVFP4vLLMmedium
    58%95% CI 53–62
    		804471552417286810061446017100945.1k2.9k · 11k103 s
    4DeepSeek V4 Flash Vision experimentalFP8 + FP4 experts · vLLM · high · 2× NVIDIA DGX Spark (GB10, 128 GB each)1 configuration1 configurationFP8 + FP4 expertsvLLMhigh
    52%95% CI 47–57
    		60505375121717849444446039904416k14k · 33k1081 s
    5MiMo V2.6 FlashFP8 + MXFP4 experts · SGLang · thinking on · 2× NVIDIA DGX Spark (GB10, 128 GB each)1 configuration1 configurationFP8 + MXFP4 expertsSGLangthinking on
    51%95% CI 46–56
    		853935451244177478563973111005614k7.1k · 33k1619 s
    6Gemma 4 31BQAT W4A16 · vLLM · thinking on · 1× NVIDIA GeForce RTX 5090 (32 GB)1 configuration1 configurationQAT W4A16vLLMthinking on
    43%95% CI 38–48
    		70505915017118989611747675392.8k2.3k · 5.3k42 s
    7Gemma 4 26B-A4BNVFP4 · vLLM · thinking on · 1× NVIDIA GeForce RTX 5090 (32 GB)1 configuration1 configurationNVFP4vLLMthinking on
    40%95% CI 35–44
    		70393515061789836111401190285.8k5.6k · 10k53 s
    8Qwen3.8 27BMLX 8-bit · MTPLX · low · Apple M5 Max (40-core GPU, 128 GB)MLX 8-bitMTPLXlow
    37%95% CI 33–42
    		7044293000176339562260680443.1k2k · 5.4k73 s
    9Qwen3.6 35B-A3BNVFP4 · vLLM · thinking on · 1× NVIDIA GeForce RTX 5090 (32 GB)1 configuration1 configurationNVFP4vLLMthinking on
    34%95% CI 29–39
    		453929150171168445028401770396.1k4.6k · 11k71 s
    score and 95% interval suite cells tinted by scoreTokens: mean output tokens per task, with the median and 90th percentile under it · Latency: median seconds per task, every attempt the score counts, which depends on the hardwarec@3: solved within three attempts, each retry shown what the deploy, tests or tools reported · Fixed: of the answers that failed the first attempt, how many a later attempt fixed (exactly how)partial not every suite run yet, so not ranked
    01

    Why intervals matter

    A few tasks going one way or the other can move a score by several points. The interval shows how much, so a two-point gap between configurations isn't sold as a win.

    Methodology →
    02

    Does quantisation hurt?

    The same model in different quantisations (10 formats, down to 2.9 bits per weight; no BF16 runs), side by side, with the interval on each.

    Quantisation study →
    03

    Is more thinking worth it?

    Score against reasoning effort, and against the output tokens that effort costs.

    Effort study →

    Private evaluations

    Want this for your team?

    Azul Labs runs Forcebench privately for two kinds of Salesforce team.

    Choosing a model to run yourself?

    A private benchmark of your shortlist

    Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

    • Private tasks written from your work only ever run on local models; they are never sent to an API.
    • Re-testing after any change of model, quantisation or engine is available.
    • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
    Get a private benchmarkHow it works

    Building or buying a Salesforce AI tool?

    A private evaluation of your tool

    Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

    • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
    • Repeat evaluations after your releases are available.
    • Typical turnaround is 2–3 weeks.
    Ask about an evaluationHow it works

    Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

    Or email leo@azl.au · azl.au