Leaderboard

Which models can actually do Salesforce work?

Two boards, ranked separately: API, models reached through a hosted service, and Local, models served on Forcebench’s own hardware. This is both boards in one table. It opens on each model at its best configuration; All configurations has one row per configuration: a model, how it was served, its quantisation, the engine that served it and the reasoning effort it ran at. Scores are the share of tasks solved, with the uncertainty drawn in.

Benchmark v0.2.0272 tasks in 15 suites42 models79 configurations18 lite effort-sweep runs not ranked hereUpdated 25 October 2026Download JSON

API: models reached through a hosted service. Local: models served on Forcebench’s own hardware.

Score by model

Each bar is the share of tasks solved within three attempts (c@3), with its 95% interval: after a failed answer, the model was shown what the deploy, tests or tools reported. Pick pass@1, the first answer alone, or one suite under Score. How it's scored

Within 3 attempts (c@3), by model

Bars whose intervals overlap are not clearly different: point at or tap one to highlight which. Ranks marked = are ties: the same score to the whole percent.

95% interval

=1Claude Opus 5.5 low · API tiebest of 3 configurations

96%

=1Gemini 4 Argon medium · API tiebest of 3 configurations

96%

3GPT-6 Astra medium · APIbest of 2 configurations

95%

=4GPT-6.1 Sol high · API tie1 configuration

93%

=4Claude Sonnet 5.5 high · API tiebest of 3 configurations

93%

6Grok 4.7 high · API1 configuration

92%

=7Grok 4.6 high · API tie1 configuration

89%

=7Gemini 3.8 Flash medium · API tie1 configuration

89%

=9GPT-5.6 Sol medium · API tie1 configuration

88%

=9Claude Fable 5.1 high · API tie1 configuration

88%

=9Claude Opus 4.8 medium · API tiebest of 4 configurations

88%

12Muse Wonder low · APIbest of 4 configurations

87%

13GPT-6 Luna medium · API1 configuration

82%

14DeepSeek V4.1 Flash high · APIbest of 3 configurations

81%

15Claude Sonnet 5 high · APIbest of 4 configurations

77%

=16Claude Haiku 5.5 medium · API tiebest of 3 configurations

76%

=16DeepSeek V4.1 Flash 1017 high · Local · Native FP8, FP4 experts tie1 configuration

76%

18MiMo V2.6 Pro thinking on · API1 configuration

71%

19DeepSeek V4 Pro high · API1 configuration

69%

20GLM-5.3 Flash high · APIbest of 5 configurations

65%

21GLM-5.3 low · API1 configuration

63%

=22Inkling Small high · API tie1 configuration

60%

=22Mistral Large 4 high · API tie1 configuration

60%

24Nemotron 3 Ultra thinking on · API1 configuration

59%

25Qwen3.8 Flash-Next medium · Local · NVFP4best of 6 configurations

58%

26MiMo V2.6 Flash thinking on · APIbest of 2 configurations

55%

27DeepSeek V4 Flash Vision experimental high · Local · FP8 + FP4 experts1 configuration

52%

28Claude Haiku 4.5 thinking on · APIbest of 2 configurations

50%

29Nemotron 3.5 Super VL thinking on · Local · NVFP4 mixed precision (NVIDIA ModelOpt)1 configuration

47%

30Muse Glimmer 30B high · Local · NVFP4 + FP8 mixed (NVIDIA ModelOpt)1 configuration

45%

31Gemma 4 31B thinking on · Local · QAT W4A161 configuration

43%

32MiniMax M3.1 Flash high · Local · NVFP4 experts, MXFP8 (QAT)1 configuration

42%

33Gemma 4 26B-A4B thinking on · Local · NVFP41 configuration

40%

=34Qwen3.8 27B low · Local · MLX 8-bit tiebest of 6 configurations

37%

=34Laguna S 2.1 thinking on · API tie1 configuration

37%

36Ling 3.0 Flash thinking on · API1 configuration

36%

37Qwen3.6 35B-A3B thinking on · Local · NVFP41 configuration

34%

38Ornith 1.5 122B thinking on · Local · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)1 configuration

33%

39MiniCPM-V 4.7 35B-A3B thinking on · Local · BF16best of 2 configurations

31%

40Nemotron 3.5 Bolt thinking on · Local · IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt)1 configuration

24%

41Nemotron 3.5 Lightning thinking on · API1 configuration

20%

Not ranked yet, waiting for their retries: Nemotron 3 Super (thinking on). Each is added once its retries finish and it has a score within three attempts.

Show the numbers

Model	Score	95% CI
Claude Opus 5.5 low · API · best of 3 configurations	96%	93–98
Gemini 4 Argon medium · API · best of 3 configurations	96%	93–98
GPT-6 Astra medium · API · best of 2 configurations	95%	92–97
GPT-6.1 Sol high · API · 1 configuration	93%	90–95
Claude Sonnet 5.5 high · API · best of 3 configurations	93%	90–96
Grok 4.7 high · API · 1 configuration	92%	89–95
Grok 4.6 high · API · 1 configuration	89%	86–92
Gemini 3.8 Flash medium · API · 1 configuration	89%	85–92
GPT-5.6 Sol medium · API · 1 configuration	88%	85–92
Claude Fable 5.1 high · API · 1 configuration	88%	85–91
Claude Opus 4.8 medium · API · best of 4 configurations	88%	84–91
Muse Wonder low · API · best of 4 configurations	87%	84–91
GPT-6 Luna medium · API · 1 configuration	82%	77–86
DeepSeek V4.1 Flash high · API · best of 3 configurations	81%	77–85
Claude Sonnet 5 high · API · best of 4 configurations	77%	73–81
Claude Haiku 5.5 medium · API · best of 3 configurations	76%	72–81
DeepSeek V4.1 Flash 1017 high · Local · Native FP8, FP4 experts · 1 configuration	76%	71–80
MiMo V2.6 Pro thinking on · API · 1 configuration	71%	66–75
DeepSeek V4 Pro high · API · 1 configuration	69%	65–74
GLM-5.3 Flash high · API · best of 5 configurations	65%	60–69
GLM-5.3 low · API · 1 configuration	63%	58–68
Inkling Small high · API · 1 configuration	60%	55–66
Mistral Large 4 high · API · 1 configuration	60%	55–65
Nemotron 3 Ultra thinking on · API · 1 configuration	59%	54–64
Qwen3.8 Flash-Next medium · Local · NVFP4 · best of 6 configurations	58%	53–62
MiMo V2.6 Flash thinking on · API · best of 2 configurations	55%	50–60
DeepSeek V4 Flash Vision experimental high · Local · FP8 + FP4 experts · 1 configuration	52%	47–57
Claude Haiku 4.5 thinking on · API · best of 2 configurations	50%	45–55
Nemotron 3.5 Super VL thinking on · Local · NVFP4 mixed precision (NVIDIA ModelOpt) · 1 configuration	47%	42–53
Muse Glimmer 30B high · Local · NVFP4 + FP8 mixed (NVIDIA ModelOpt) · 1 configuration	45%	40–50
Gemma 4 31B thinking on · Local · QAT W4A16 · 1 configuration	43%	38–48
MiniMax M3.1 Flash high · Local · NVFP4 experts, MXFP8 (QAT) · 1 configuration	42%	37–47
Gemma 4 26B-A4B thinking on · Local · NVFP4 · 1 configuration	40%	35–44
Qwen3.8 27B low · Local · MLX 8-bit · best of 6 configurations	37%	33–42
Laguna S 2.1 thinking on · API · 1 configuration	37%	32–42
Ling 3.0 Flash thinking on · API · 1 configuration	36%	32–41
Qwen3.6 35B-A3B thinking on · Local · NVFP4 · 1 configuration	34%	29–39
Ornith 1.5 122B thinking on · Local · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · 1 configuration	33%	28–38
MiniCPM-V 4.7 35B-A3B thinking on · Local · BF16 · best of 2 configurations	31%	26–36
Nemotron 3.5 Bolt thinking on · Local · IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · 1 configuration	24%	20–28
Nemotron 3.5 Lightning thinking on · API · 1 configuration	20%	16–24

Breakdown by suite

Each suite's score on the page's metric, the same rows in the same order as the chart above: within three attempts (c@3) while the page is on c@3, the first answer (pass@1) when you pick pass@1 under Score. Pick a suite's name to rank the page by it.

c@3 by suite

Darker is higher on light backgrounds, brighter is higher on dark ones; every cell prints its score.

c@3 on each suite, in percent, for each model at its best configuration, in the order of the chart above. A dash: no complete score on that suite.
Model
Claude Opus 5.5low · API	100	83	94	100	82	100	100	100	100	94	94	100	94	100	94
Gemini 4 Argonmedium · API	95	94	100	100	76	100	94	100	100	100	94	93	89	100	100
GPT-6 Astramedium · API	100	89	100	100	82	94	100	100	100	94	100	100	67	100	94
GPT-6.1 Solhigh · API	100	89	94	100	76	100	100	100	100	78	94	100	61	100	100
Claude Sonnet 5.5high · API	100	89	100	100	76	89	100	95	100	94	83	93	78	100	94
Grok 4.7high · API	100	89	88	100	65	94	100	95	100	94	83	93	83	100	94
Grok 4.6high · API	100	78	94	100	53	94	100	89	100	100	83	93	61	100	89
Gemini 3.8 Flashmedium · API	100	89	88	100	47	100	89	95	100	100	78	87	61	100	94
GPT-5.6 Solmedium · API	95	78	88	100	53	61	100	95	100	94	89	93	78	100	100
Claude Fable 5.1high · API	100	39	94	85	59	100	100	100	100	94	83	93	83	95	94
Claude Opus 4.8medium · API	100	94	100	100	53	94	100	95	94	89	83	93	50	100	72
Muse Wonderlow · API	95	83	88	100	35	89	83	100	100	100	89	93	67	95	94
GPT-6 Lunamedium · API	85	72	88	100	47	61	94	84	100	83	78	87	56	100	89
DeepSeek V4.1 Flashhigh · API	95	83	88	95	29	67	61	89	100	89	78	87	61	95	94
Claude Sonnet 5high · API	90	89	76	90	24	67	100	79	100	83	72	80	50	100	56
Claude Haiku 5.5medium · API	90	89	94	85	29	61	78	95	94	83	67	87	50	95	50
DeepSeek V4.1 Flash 1017high · Local · Native FP8, FP4 experts	90	72	82	95	18	56	72	84	100	72	67	87	56	95	94
MiMo V2.6 Prothinking on · API	90	72	71	90	35	39	28	95	94	78	61	80	39	100	89
DeepSeek V4 Prohigh · API	80	67	88	90	18	56	17	89	100	72	50	87	39	100	89
GLM-5.3 Flashhigh · API	90	72	82	75	12	39	22	79	94	67	44	80	44	90	78
GLM-5.3low · API	75	56	88	55	24	39	50	74	100	61	50	73	28	90	83
Inkling Smallhigh · API	80	56	65	65	24	17	44	84	100	56	56	67	33	95	67
Mistral Large 4high · API	75	39	82	70	18	50	28	74	94	67	39	67	39	100	61
Nemotron 3 Ultrathinking on · API	75	61	71	60	24	33	22	79	94	56	39	67	28	95	83
Qwen3.8 Flash-Nextmedium · Local · NVFP4	80	44	71	55	24	17	28	68	100	61	44	60	17	100	94
MiMo V2.6 Flashthinking on · API	90	44	65	55	24	22	17	68	94	56	44	73	11	100	56
DeepSeek V4 Flash Vision experimentalhigh · Local · FP8 + FP4 experts	60	50	53	75	12	17	17	84	94	44	44	60	39	90	44
Claude Haiku 4.5thinking on · API	75	44	65	60	0	44	17	79	89	56	33	73	11	70	28
Nemotron 3.5 Super VLthinking on · Local · NVFP4 mixed precision (NVIDIA ModelOpt)	60	50	53	40	24	17	22	79	83	61	33	53	11	85	39
Muse Glimmer 30Bhigh · Local · NVFP4 + FP8 mixed (NVIDIA ModelOpt)	75	39	53	30	12	17	22	84	83	56	22	47	11	80	39
Gemma 4 31Bthinking on · Local · QAT W4A16	70	50	59	15	0	17	11	89	89	61	17	47	6	75	39
MiniMax M3.1 Flashhigh · Local · NVFP4 experts, MXFP8 (QAT)	55	33	35	40	12	0	17	68	83	50	28	67	11	85	50
Gemma 4 26B-A4Bthinking on · Local · NVFP4	70	39	35	15	0	6	17	89	83	61	11	40	11	90	28
Qwen3.8 27Blow · Local · MLX 8-bit	70	44	29	30	0	0	17	63	39	56	22	60	6	80	44
Laguna S 2.1thinking on · API	55	33	41	30	6	17	17	53	67	39	22	47	0	90	39
Ling 3.0 Flashthinking on · API	75	33	35	15	0	22	11	74	72	44	17	33	11	75	28
Qwen3.6 35B-A3Bthinking on · Local · NVFP4	45	39	29	15	0	17	11	68	44	50	28	40	17	70	39
Ornith 1.5 122Bthinking on · Local · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)	40	28	35	35	0	11	11	58	72	39	28	40	11	65	17
MiniCPM-V 4.7 35B-A3Bthinking on · Local · BF16	50	33	35	20	18	6	11	53	44	50	17	33	11	65	22
Nemotron 3.5 Boltthinking on · Local · IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt)	40	22	29	5	0	0	11	58	6	50	17	20	6	80	17
Nemotron 3.5 Lightningthinking on · API	15	33	18	10	0	6	11	47	11	39	11	7	6	65	22

Every number

How to read this. Best per model shows each model's highest-scoring configuration. Models run in more configurations get more chances at a high score, so check the N badge. API and local configurations in one ranking. Scores are within three attempts (c@3): after a failed answer, the model is shown what the deploy, tests or tools reported and answers again. Each has a 95% confidence interval. A configuration still waiting for its retries is listed but not ranked yet, and pass@1, the first answer alone, is a choice away under Score. The dot is the score and the bar the interval. Where two configurations’ bars overlap, they are not clearly different; where they don’t, one is clearly above the other. Latency is not comparable between API and local configurations, or across hardware. How the best local configuration compares with the API ones: local vs API. Methodology

Claude Opus 5.5API · low

API

Not disclosed

—

low

9693–98

100

1.2k

9.8 s

Every configuration of Claude Opus 5.5 on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · low	96%95% CI 93–98	1.2k	best, shown above
API · high	95%95% CI 93–98	2.4k	not clearly different
API · medium	95%95% CI 92–97	1.8k	not clearly different

Claude Opus 5.5: every configuration, on every board →

Gemini 4 ArgonAPI · medium

API

Not disclosed

—

medium

9693–98

100

708

10 s

Every configuration of Gemini 4 Argon on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · medium	96%95% CI 93–98	708	best, shown above
API · low	95%95% CI 92–97	598	not clearly different
API · high	95%95% CI 92–97	5.3k	not clearly different

Gemini 4 Argon: every configuration, on every board →

GPT-6 AstraAPI · medium

API

Not disclosed

—

medium

9592–97

100

846

7.4 s

Every configuration of GPT-6 Astra on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · medium	95%95% CI 92–97	846	best, shown above
API · high	92%95% CI 90–95	1.8k	not clearly different

GPT-6 Astra: every configuration, on every board →

GPT-6.1 SolAPI · high1 configuration1 configuration

API

Not disclosed

—

high

9390–95

100

1.5k

20 s

Claude Sonnet 5.5API · high

API

Not disclosed

—

high

9390–96

100

1.8k

9.3 s

Every configuration of Claude Sonnet 5.5 on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · high	93%95% CI 90–96	1.8k	best, shown above
API · low	90%95% CI 86–93	1k	not clearly different
API · medium	88%95% CI 85–92	1.1k	not clearly different

Claude Sonnet 5.5: every configuration, on every board →

Grok 4.7API · high1 configuration1 configuration

API

Not disclosed

—

high

9289–95

100

13k

111 s

Grok 4.6API · high1 configuration1 configuration

API

Not disclosed

—

high

8986–92

100

6.2k

69 s

Gemini 3.8 FlashAPI · medium1 configuration1 configuration

API

Not disclosed

—

medium

8985–92

100

7.4k

41 s

GPT-5.6 SolAPI · medium1 configuration1 configuration

API

Not disclosed

—

medium

8885–92

100

1.4k

10 s

Claude Fable 5.1API · high1 configuration1 configuration

API

Not disclosed

—

high

8885–91

100

2.1k

18 s

Claude Opus 4.8API · medium

API

Not disclosed

—

medium

8884–91

100

1.6k

11 s

Every configuration of Claude Opus 4.8 on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · medium	88%95% CI 84–91	1.6k	best, shown above
API · high	87%95% CI 83–91	2k	not clearly different
API · low	85%95% CI 81–89	940	not clearly different
API · thinking off	82%95% CI 78–86	882	not clearly different

Claude Opus 4.8: every configuration, on every board →

Muse WonderAPI · low

API

Not disclosed

—

low

8784–91

100

15 s

Every configuration of Muse Wonder on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · low	87%95% CI 84–91	2k	best, shown above
API · medium	87%95% CI 84–91	3.1k	not clearly different
API · max	87%95% CI 84–91	8.2k	not clearly different
API · high	86%95% CI 82–89	4.4k	not clearly different

Muse Wonder: every configuration, on every board →

GPT-6 LunaAPI · medium1 configuration1 configuration

API

Not disclosed

—

medium

8277–86

100

1.5k

18 s

DeepSeek V4.1 FlashAPI · FP8 · highopen weights

API

FP8

high

8177–85

100

9.6k

136 s

Every configuration of DeepSeek V4.1 Flash on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · FP8 · high	81%95% CI 77–85	9.6k	best, shown above
Local · EXL3 2.9bpw · vLLM + ExLlamaV3 · high	73%95% CI 68–78	7.4k	not clearly different
Local · Native FP8, FP4 experts · SGLang · high	70%95% CI 66–75	9.6k	clearly lower

DeepSeek V4.1 Flash: every configuration, on every board →

Claude Sonnet 5API · high

API

Not disclosed

—

high

7773–81

100

2.4k

11 s

Every configuration of Claude Sonnet 5 on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · high	77%95% CI 73–81	2.4k	best, shown above
API · medium	75%95% CI 71–79	1.2k	not clearly different
API · thinking off	71%95% CI 67–76	919	not clearly different
API · low	69%95% CI 64–74	730	not clearly different

Claude Sonnet 5: every configuration, on every board →

=16

Claude Haiku 5.5API · medium

API

Not disclosed

—

medium

7672–81

2.8k

8.7 s

Every configuration of Claude Haiku 5.5 on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · medium	76%95% CI 72–81	2.8k	best, shown above
API · high	76%95% CI 72–80	4.5k	not clearly different
API · low	74%95% CI 69–78	2.2k	not clearly different

Claude Haiku 5.5: every configuration, on every board →

=16

DeepSeek V4.1 Flash 1017Local · Native FP8, FP4 experts · SGLang · high1 configuration1 configurationopen weights

Local

Native FP8, FP4 experts

SGLang

high

7671–80

100

8.9k

454 s

MiMo V2.6 ProAPI · FP8 · thinking on1 configuration1 configurationopen weights

API

FP8

thinking on

7166–75

100

14k

285 s

DeepSeek V4 ProAPI · FP8 · high1 configuration1 configurationopen weights

API

FP8

high

6965–74

100

11k

127 s

GLM-5.3 FlashAPI · FP8 · highopen weights

API

FP8

high

6560–69

3.5k

30 s

Every configuration of GLM-5.3 Flash on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · FP8 · high	65%95% CI 60–69	3.5k	best, shown above
Local · EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · high	64%95% CI 59–68	3.4k	not clearly different
Local · EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · low	60%95% CI 55–66	733	not clearly different
Local · EXL3 4.0bpw experts, 4-bit dense · TensorFold · high	54%95% CI 49–59	4k	clearly lower
Local · EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off	50%95% CI 44–55	2.3k	clearly lower

GLM-5.3 Flash: every configuration, on every board →

GLM-5.3API · FP8 · low1 configuration1 configurationopen weights

API

FP8

low

6358–68

100

753

13 s

=22

Inkling SmallAPI · FP8 · high1 configuration1 configurationopen weights

API

FP8

high

6055–66

100

5.4k

58 s

=22

Mistral Large 4API · high1 configuration1 configuration

API

Not disclosed

—

high

6055–65

100

15k

219 s

Nemotron 3 UltraAPI · thinking on1 configuration1 configurationopen weights

API

Not disclosed

—

thinking on

5954–64

1.8k

55 s

Qwen3.8 Flash-NextLocal · NVFP4 · vLLM · mediumopen weights

Local

NVFP4

vLLM

medium

5853–62

100

5.1k

103 s

Every configuration of Qwen3.8 Flash-Next on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
Local · NVFP4 · vLLM · medium	58%95% CI 53–62	5.1k	best, shown above
Local · MLX 4-bit · TensorFold · medium	51%95% CI 46–56	12k	not clearly different
Local · MLX 4-bit · MTPLX · medium	50%95% CI 45–55	9.8k	not clearly different
Local · INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · medium	49%95% CI 43–54	13k	not clearly different
Local · INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · medium	46%95% CI 41–51	13k	clearly lower
API · thinking on	44%95% CI 39–49	21k	clearly lower

Qwen3.8 Flash-Next: every configuration, on every board →

MiMo V2.6 FlashAPI · FP8 · thinking onopen weights

API

FP8

thinking on

5550–60

100

13k

244 s

Every configuration of MiMo V2.6 Flash on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · FP8 · thinking on	55%95% CI 50–60	13k	best, shown above
Local · FP8 + MXFP4 experts · SGLang · thinking on	51%95% CI 46–56	14k	not clearly different

MiMo V2.6 Flash: every configuration, on every board →

DeepSeek V4 Flash Vision experimentalLocal · FP8 + FP4 experts · vLLM · high1 configuration1 configurationopen weights

Local

FP8 + FP4 experts

vLLM

high

5247–57

16k

1081 s

Claude Haiku 4.5API · thinking on

API

Not disclosed

—

thinking on

5045–55

5.1k

41 s

Every configuration of Claude Haiku 4.5 on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
API · thinking on	50%95% CI 45–55	5.1k	best, shown above
API · thinking off	46%95% CI 41–52	1.1k	not clearly different

Claude Haiku 4.5: every configuration, on every board →

Nemotron 3.5 Super VLLocal · NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on1 configuration1 configurationopen weights

Local

NVFP4 mixed precision (NVIDIA ModelOpt)

TensorFold (Zig)

thinking on

4742–53

10k

953 s

Muse Glimmer 30BLocal · NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · high1 configuration1 configurationopen weights

Local

NVFP4 + FP8 mixed (NVIDIA ModelOpt)

vLLM

high

4540–50

111 s

Gemma 4 31BLocal · QAT W4A16 · vLLM · thinking on1 configuration1 configurationopen weights

Local

QAT W4A16

vLLM

thinking on

4338–48

2.8k

42 s

MiniMax M3.1 FlashLocal · NVFP4 experts, MXFP8 (QAT) · vLLM · high1 configuration1 configurationopen weights

Local

NVFP4 experts, MXFP8 (QAT)

vLLM

high

4237–47

9.6k

958 s

Gemma 4 26B-A4BLocal · NVFP4 · vLLM · thinking on1 configuration1 configurationopen weights

Local

NVFP4

vLLM

thinking on

4035–44

5.8k

53 s

=34

Qwen3.8 27BLocal · MLX 8-bit · MTPLX · lowopen weights

Local

MLX 8-bit

MTPLX

low

3733–42

3.1k

73 s

Every configuration of Qwen3.8 27B on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
Local · MLX 8-bit · MTPLX · low	37%95% CI 33–42	3.1k	best, shown above
Local · AWQ-INT4 · vLLM · low	37%95% CI 32–42	3.3k	not clearly different
Local · Splash 4-bit · Splash · low	36%95% CI 31–41	3.4k	not clearly different
Local · MLX 4-bit · MTPLX · low	35%95% CI 30–41	3.4k	not clearly different
Local · AWQ-INT4 · vLLM · xhigh	35%95% CI 30–40	18k	not clearly different
Local · AWQ-INT4 · vLLM · medium	33%95% CI 29–38	4.7k	not clearly different

Qwen3.8 27B: every configuration, on every board →

=34

Laguna S 2.1API · FP4 · thinking on1 configuration1 configurationopen weights

API

FP4

thinking on

3732–42

3.4k

30 s

Ling 3.0 FlashAPI · BF16 · thinking on1 configuration1 configurationopen weights

API

BF16

thinking on

3632–41

4.9k

101 s

Qwen3.6 35B-A3BLocal · NVFP4 · vLLM · thinking on1 configuration1 configurationopen weights

Local

NVFP4

vLLM

thinking on

3429–39

6.1k

71 s

Ornith 1.5 122BLocal · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on1 configuration1 configurationopen weights

Local

NVFP4 experts, FP8 attention (NVIDIA ModelOpt)

vLLM

thinking on

3328–38

13k

2007 s

MiniCPM-V 4.7 35B-A3BLocal · BF16 · vLLM · thinking onopen weights

Local

BF16

vLLM

thinking on

3126–36

1.9k

219 s

Every configuration of MiniCPM-V 4.7 35B-A3B on this board, against its best
Configuration	Within 3 attempts c@3 · 95% CI	Tokens	Against the best 95% intervals
Local · BF16 · vLLM · thinking on	31%95% CI 26–36	1.9k	best, shown above
Local · INT4 W4A16 experts, BF16 dense · vLLM · thinking on	28%95% CI 23–33	2.4k	not clearly different

MiniCPM-V 4.7 35B-A3B: every configuration, on every board →

Nemotron 3.5 BoltLocal · IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on1 configuration1 configurationopen weights

Local

IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt)

TensorFold (Zig)

thinking on

2420–28

15k

2098 s

Nemotron 3.5 LightningAPI · NVFP4 · thinking on1 configuration1 configurationopen weights

API

NVFP4

thinking on

2016–24

6.6k

447 s

–

Nemotron 3 SuperAPI · thinking onopen weightsretries pending

API

Not disclosed

—

thinking on

—

2.6k

—

score and 95% interval suite cells tinted by scoreTokens: mean output tokens per task · Latency: median seconds per task, every attempt the score countsc@3: solved within three attempts, each retry shown what the deploy, tests or tools reported · Fixed: of the answers that failed the first attempt, how many a later attempt fixed (exactly how)partial not every suite run yet, so not ranked

Why intervals matter

A few tasks going one way or the other can move a score by several points. The interval shows how much, so a two-point gap between configurations isn't sold as a win.

Methodology →

Does quantisation hurt?

The same model in different quantisations (12 formats, down to 2.9 bits per weight), side by side, with the interval on each.

Quantisation study →

Is more thinking worth it?

Score against reasoning effort, and against the output tokens that effort costs.

Effort study →

Private evaluations

Want this for your team?

Azul Labs runs Forcebench privately for two kinds of Salesforce team.

Choosing a model to run yourself?

A private benchmark of your shortlist

Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

Private tasks written from your work only ever run on local models; they are never sent to an API.
Re-testing after any change of model, quantisation or engine is available.
Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.

Get a private benchmark How it works

Building or buying a Salesforce AI tool?

A private evaluation of your tool

Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
Repeat evaluations after your releases are available.
Typical turnaround is 2–3 weeks.

Ask about an evaluation How it works

Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

Or email leo@azl.au · azl.au