Models / claude-sonnet-5-5

Claude Sonnet 5.5

Anthropic's model, with closed weights. Forcebench has run it in 3 configurations on the API board, plus 3 on the lite subset. API is models reached through a hosted service; Local, models served on Forcebench’s own hardware.

On the API board

Models reached through a hosted service. Claude Sonnet 5.5 is ranked by the highest-scoring of its 3 configurations: the joint third-highest score of 26 models, not clearly different from 9 other models. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board → Cost and speed →

93%
within 3 attempts (c@3), 95% CI 90–96 · high · joint third-highest score of 26 models, not clearly different from 9 other models
1.8k
output tokens per task · median 1.2k, p90 4.1k
9.3 s
median per task, every attempt counted, including network and queueing · 90th percentile 33 s
0%
no answer within the 32,768-token budget
$0.026
per task at the list price of 10 October 2026, every attempt counted
$0.028
per solved task: the total cost over the tasks solved · cost and speed
Every configuration
Within 3 attempts (c@3) with 95% intervals, highest first. Bars whose intervals overlap are not clearly different.
  • 95% interval
high Anthropichighest score
93%, 95% interval 90–96%
low Anthropic
90%, 95% interval 86–93%
medium Anthropic
88%, 95% interval 85–92%
    Show the numbers
    Every configuration of Claude Sonnet 5.5 on the API board
    ConfigurationRank all configsWithin 3 attempts c@3 · 95% CIAgainst the highest by 95% interval overlapTokens mean · median · p90No answerLatency median per task, every attempt countedCost per taskCost per solved task
    Anthropic · highhighest score5
    93%95% CI 90–96
    		—1.8k1.2k · 4.1k0%9.3 s$0.026$0.028
    Anthropic · low9
    90%95% CI 86–93
    		not clearly different1k699 · 2.2k0%5.9 s$0.017$0.019
    Anthropic · medium12
    88%95% CI 85–92
    		not clearly different1.1k747 · 2.7k0%6.1 s$0.019$0.021
    Suites, at its highest-scoring configuration
    high · Anthropic: within 3 attempts (c@3) on each suite with 95% intervals, against the median of the API board's models, each at its highest-scoring configuration.
    • 95% interval
    • API board median
    Apex
    100%, 95% interval 84–100%
    Salesforce APIs
    89%, 95% interval 67–97%
    CI/CD
    100%, 95% interval 82–100%
    Salesforce CLI
    100%, 95% interval 84–100%
    Salesforce docs
    76%, 95% interval 53–90%
    fflib Enterprise Patterns
    89%, 95% interval 67–97%
    Flow
    100%, 95% interval 82–100%
    Governor limits & pushback
    95%, 95% interval 75–99%
    Lightning Web Components
    100%, 95% interval 82–100%
    Nonprofit Success Pack
    94%, 95% interval 74–99%
    Packaging
    83%, 95% interval 61–94%
    Permissions & access
    93%, 95% interval 70–99%
    Scratch org definitions
    78%, 95% interval 55–91%
    SOQL
    100%, 95% interval 84–100%
    Trigger Actions Framework
    94%, 95% interval 74–99%
      Show the numbers
      Per-suite scores of Claude Sonnet 5.5's highest-scoring configuration on the API board, with the board's median
      Suitec@3 95% CIAPI medianTasks
      Apex
      100%95% CI 84–100
      		90%20
      Salesforce APIs
      89%95% CI 67–97
      		72%18
      CI/CD
      100%95% CI 82–100
      		88%17
      Salesforce CLI
      100%95% CI 84–100
      		90%20
      Salesforce docs
      76%95% CI 53–90
      		29%17
      fflib Enterprise Patterns
      89%95% CI 67–97
      		61%18
      Flow
      100%95% CI 82–100
      		69%18
      Governor limits & pushback
      95%95% CI 75–99
      		89%19
      Lightning Web Components
      100%95% CI 82–100
      		100%18
      Nonprofit Success Pack
      94%95% CI 74–99
      		81%18
      Packaging
      83%95% CI 61–94
      		69%18
      Permissions & access
      93%95% CI 70–99
      		87%15
      Scratch org definitions
      78%95% CI 55–91
      		50%18
      SOQL
      100%95% CI 84–100
      		100%20
      Trigger Actions Framework
      94%95% CI 74–99
      		86%18

      By reasoning effort

      The same configuration at several efforts, each against its lowest reasoning effort on the same tasks (a paired 95% interval), and charted against the effort tier and against the output tokens it costs. These scores are pass@1, the first answer alone: a paired comparison needs both efforts' answers to the same tasks, and not every effort has had its retries. Lite-subset sweeps (4 tasks per suite) are never plotted with the full benchmark. The effort study has every model.

      Claude Sonnet 5.5 by reasoning effort, full benchmark: each effort against the sweep's lowest reasoning effort
      Configuration · effort · full benchmarkpass@1 95% CIAgainst the lowest reasoning effort same tasks, pairedTokens
      lowbaseline
      77%95% CI 73–82
      		—1k
      medium
      75%95% CI 70–79
      		no clear difference (−2 pts, 95% CI −6 pts to +0.8 pts); this data can't rule out a change of up to 6 points1.1k
      high
      82%95% CI 77–86
      		clearly higher (+5 pts, 95% CI +0.1 pts to +9 pts)1.8k
      Score by effort tier · full benchmark
      Overall pass@1; vertical bars are 95% confidence intervals.
      Claude Sonnet 5.5, full benchmark: pass@1 by reasoning effort tierClaude Sonnet 5.5: low 77%, medium 75%, high 82%0%25%50%75%100%LowMediumHighReasoning effort tierpass@1Claude Sonnet 5.5
      Show the numbers
      ModelEffort (tier)pass@195% CITasksNo answerTokens: meanmedianLatency: median per answer
      Claude Sonnet 5.5low77%73–822720%1k6995.6 s
      Claude Sonnet 5.5medium75%70–792720%1.1k7475.6 s
      Claude Sonnet 5.5high · vendor default82%77–862720%1.8k1.2k9.0 s
      Score against output tokens · full benchmark
      Each point is one effort setting. Further right means more output tokens per task. The numbers are under the chart above.
      Claude Sonnet 5.5, full benchmark: pass@1 against mean output tokens per taskClaude Sonnet 5.5: 1k tokens 77%, 1.1k tokens 75%, 1.8k tokens 82%0%25%50%75%100%1k2kMean output tokens per task (log scale)pass@1lowmediumhighClaude Sonnet 5.5
      Claude Sonnet 5.5 by reasoning effort, lite subset: each effort against the sweep's lowest reasoning effort
      Configuration · effort · lite subsetpass@1 95% CIAgainst the lowest reasoning effort same tasks, pairedTokens
      lowbaseline
      73%95% CI 63–82
      		—1k
      medium
      77%95% CI 68–85
      		no clear difference (+4 pts, 95% CI −2 pts to +10 pts); this data can't rule out a change of up to 10 points1.1k
      max
      45%95% CI 37–53
      		clearly lower (−28 pts, 95% CI −38 pts to −18 pts)21k
      Score by effort tier · lite subset
      Overall pass@1; vertical bars are 95% confidence intervals.
      Claude Sonnet 5.5, lite subset: pass@1 by reasoning effort tierClaude Sonnet 5.5: low 73%, medium 77%, max 45%0%25%50%75%100%LowMediumMaxReasoning effort tierpass@1Claude Sonnet 5.5
      Show the numbers
      ModelEffort (tier)pass@195% CITasksNo answerTokens: meanmedianLatency: median per answer
      Claude Sonnet 5.5low73%63–82600%1k7536.1 s
      Claude Sonnet 5.5medium77%68–85600%1.1k9196.5 s
      Claude Sonnet 5.5max45%37–536048%21k29k201 s
      Score against output tokens · lite subset
      Each point is one effort setting. Further right means more output tokens per task. The numbers are under the chart above.
      Claude Sonnet 5.5, lite subset: pass@1 against mean output tokens per taskClaude Sonnet 5.5: 1k tokens 73%, 1.1k tokens 77%, 21k tokens 45%0%25%50%75%100%1k2k5k10k20kMean output tokens per task (log scale)pass@1lowmediummaxClaude Sonnet 5.5

      In the studies

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au