Study · Max effort
Max effort, under a fixed output budget
Vendors describe max as reasoning with no limit on token spending, and advise a larger output budget at the top settings than the 32,768 tokens Forcebench gives every answer. Forcebench keeps that budget on purpose, so results stay comparable across models and efforts. So this study measures max under Forcebench’s budget, not max as vendors intend it: every run at max is compared with the same model, quantisation and engine one level down, on exactly the same tasks.
Of 7 configurations run at max, 3 scored clearly lower and 4 showed no clear difference against one level down (paired 95% intervals). Up to 55% of max answers reached the 32,768-token budget before answering, and count as failed. On only the tasks max answered, 1 scored clearly higher and 6 showed no clear difference; those tasks are chosen by max's own answers, so that comparison leans towards max.
- Claude Sonnet 5.5
- Claude Haiku 5.5
- GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense
- DeepSeek V4.1 Flash · Native FP8, FP4 experts
- Qwen3.8 27B · AWQ-INT4
- DeepSeek V4.1 Flash · EXL3 2.9bpw
- GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit dense
Show the numbers
| Model | Tasks | Compared with | Change at max (paired) | 95% CI | No answer at max | Answered-only change | Answered tasks | Mean output tokens (below → max) | Median latency at max |
|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 60 | high | −35 pts | −45 pts to −25 pts | 48% | ±0 pts (±0 pts to ±0 pts) | 31 | 1.8k → 21k | 201 s |
| Claude Haiku 5.5 | 60 | high | −22 pts | −32 pts to −10 pts | 55% | +9 pts (−5 pts to +23 pts) | 27 | 4.5k → 25k | 129 s |
| GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense | 60 | high | −13 pts | −25 pts to −2 pts | 27% | −9 pts (−23 pts to +4 pts) | 44 | 3.4k → 15k | 688 s |
| DeepSeek V4.1 Flash · Native FP8, FP4 experts | 60 | high | −3 pts | −13 pts to +7 pts | 27% | +5 pts (−7 pts to +15 pts) | 44 | 9.6k → 16k | 527 s |
| Qwen3.8 27B · AWQ-INT4 | 272 | medium | −3 pts | −6 pts to +1 pt | 25% | −1 pt (−6 pts to +3 pts) | 200 | 4.7k → 18k | 141 s |
| DeepSeek V4.1 Flash · EXL3 2.9bpw | 60 | high | ±0 pts | −12 pts to +12 pts | 30% | +22 pts (+13 pts to +30 pts) | 42 | 7.4k → 18k | 1356 s |
| GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit dense | 60 | high | +1 pt | −8 pts to +13 pts | 28% | +7 pts (−7 pts to +20 pts) | 43 | 4k → 16k | 570 s |
high → max: −35 pts
On the same 60 tasks: clearly lower (−35 pts, 95% CI −45 pts to −25 pts). 48% of the max answers reached the 32,768-token budget before answering. On the 31 tasks max answered: no task's grade changed. Mean output tokens went from 1.8k to 21k.
high → max: −22 pts
On the same 60 tasks: clearly lower (−22 pts, 95% CI −32 pts to −10 pts). 55% of the max answers reached the 32,768-token budget before answering. On the 27 tasks max answered: no clear difference (+9 pts, 95% CI −5 pts to +23 pts); this data can't rule out a change of up to 23 points, from only 27 tasks, so the interval is wide. Mean output tokens went from 4.5k to 25k.
high → max: −13 pts
On the same 60 tasks: clearly lower (−13 pts, 95% CI −25 pts to −2 pts). 27% of the max answers reached the 32,768-token budget before answering. On the 44 tasks max answered: no clear difference (−9 pts, 95% CI −23 pts to +4 pts); this data can't rule out a change of up to 23 points. Mean output tokens went from 3.4k to 15k.
high → max: −3 pts
On the same 60 tasks: no clear difference (−3 pts, 95% CI −13 pts to +7 pts); this data can't rule out a change of up to 13 points. 27% of the max answers reached the 32,768-token budget before answering. On the 44 tasks max answered: no clear difference (+5 pts, 95% CI −7 pts to +15 pts); this data can't rule out a change of up to 15 points. Mean output tokens went from 9.6k to 16k.
medium → xhigh: −3 pts
On the same 272 tasks: no clear difference (−3 pts, 95% CI −6 pts to +1 pt); this data can't rule out a change of up to 6 points. 25% of the max answers reached the 32,768-token budget before answering. On the 200 tasks max answered: no clear difference (−1 pt, 95% CI −6 pts to +3 pts); this data can't rule out a change of up to 6 points. Mean output tokens went from 4.7k to 18k.
high → max: ±0 pts
On the same 60 tasks: no clear difference (±0 pts, 95% CI −12 pts to +12 pts); this data can't rule out a change of up to 12 points. 30% of the max answers reached the 32,768-token budget before answering. On the 42 tasks max answered: clearly higher (+22 pts, 95% CI +13 pts to +30 pts). Mean output tokens went from 7.4k to 18k.
high → max: +1 pt
On the same 60 tasks: no clear difference (+1 pt, 95% CI −8 pts to +13 pts); this data can't rule out a change of up to 13 points. 28% of the max answers reached the 32,768-token budget before answering. On the 43 tasks max answered: no clear difference (+7 pts, 95% CI −7 pts to +20 pts); this data can't rule out a change of up to 20 points. Mean output tokens went from 4k to 16k.
What happens at max
Every answer gets a 32,768-token output budget, reasoning included. One level below max, the median answer here took 1.1k–6.1k output tokens. At max, a model often keeps reasoning until the budget runs out and never starts the answer, which scores as a failure.
Reading the traces of GLM-5.3 Flash's max run (lite subset, 30 September 2026): each of its 16 unanswered tasks stopped at exactly 32,768 tokens, all of it reasoning, before any answer, around 130,000–147,000 characters of thinking per task. In that run the reasoning was coherent and on topic: it kept re-checking Salesforce details it had already settled, with hundreds of turns of second-guessing per task, mostly without repeating itself. That is what this one run showed; whether a bigger budget would have let it finish wasn't tested.
It also happened with Claude Sonnet 5.5 on Anthropic's API, the one vendor's API run at max here: 48% of its max answers reached the budget, each stopped at exactly 32,768 tokens.
What the vendors say
Anthropic's effort docs describe max as maximum capability with no constraints on token spending, and give each model its own advice:
- For Claude Sonnet 5.5, a max_tokens of 128,000 (its maximum) for agentic coding, and xhigh or max only where your own evals show a quality gain.
- For Claude Opus 4.7 (and Opus 4.8), that max adds significant cost for relatively small quality gains on most workloads, and can lead to overthinking on some tasks.
- For others, including Claude Opus 5.5 and Fable 5.1, a large max_tokens at the higher effort levels.
Forcebench keeps one budget for every model and effort so results stay comparable; see Output budget.
What to take from it
Under a 32,768-token budget, max often ran out of room before answering, and that cost it tasks here. On the tasks it did answer, 1 scored clearly higher and 6 showed no clear difference against one level down. If you raise effort to max, give the model the larger output budget its vendor advises; this study doesn't measure max with one.
Most max runs here are on the lite subset (4 tasks per suite), so their intervals are wide. Each comparison uses exactly the same tasks at both efforts.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au