Suites / docs

Salesforce docs

Navigating the official Salesforce documentation: limits, allocations and platform behaviour an engineer has to look up while designing a solution. Each answer is a short, exact fact and must cite the developer.salesforce.com page that states it; both the fact and the cited page are checked deterministically.

17
tasks
Deterministic validators
grading
88%
highest c@3 · Claude Opus 5.5
78
configurations with results

How it's graded

Each answer is a short, exact fact and must cite the developer.salesforce.com page that states it. Both the fact and the cited page are checked deterministically.

Difficulty mix

5 easy7 medium5 hard

Leaderboard

Who's best at Salesforce docs

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 17 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at Salesforce docs
Within three attempts (c@3) on Salesforce docs, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
1Claude Opus 5.5 high
88%, 95% interval 66–97%
2GPT-6 Astra medium
82%, 95% interval 59–94%
3Gemini 4 Argon low
76%, 95% interval 53–90%
4Grok 4.7 high
65%, 95% interval 41–83%
5Muse Wonder medium
53%, 95% interval 31–74%
6MiMo V2.6 Pro thinking on
35%, 95% interval 17–59%
7DeepSeek V4.1 Flash high
29%, 95% interval 13–53%
=8GLM-5.3 low tie
24%, 95% interval 10–47%
=8Nemotron 3 Ultra thinking on tie
24%, 95% interval 10–47%
=8Inkling Small high tie
24%, 95% interval 10–47%
11Mistral Large 4 high
18%, 95% interval 6–41%
12Laguna S 2.1 thinking on
6%, 95% interval 1–27%
=13Ling 3.0 Flash thinking on tie
0%, 95% interval 0–18%
=13Qwen3.8 Flash-Next thinking on tie
0%, 95% interval 0–18%
    Show the numbers
    ModelScore95% CI
    Claude Opus 5.5 high88%66–97
    GPT-6 Astra medium82%59–94
    Gemini 4 Argon low76%53–90
    Grok 4.7 high65%41–83
    Muse Wonder medium53%31–74
    MiMo V2.6 Pro thinking on35%17–59
    DeepSeek V4.1 Flash high29%13–53
    GLM-5.3 low24%10–47
    Nemotron 3 Ultra thinking on24%10–47
    Inkling Small high24%10–47
    Mistral Large 4 high18%6–41
    Laguna S 2.1 thinking on6%1–27
    Ling 3.0 Flash thinking on0%0–18
    Qwen3.8 Flash-Next thinking on0%0–18
    Local: the best of each lab at Salesforce docs
    Within three attempts (c@3) on Salesforce docs, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    =1GLM-5.3 Flash low · EXL3 4.0bpw experts, FP8 dense tie
    24%, 95% interval 10–47%
    =1Qwen3.8 Flash-Next medium · NVFP4 tie
    24%, 95% interval 10–47%
    =1DeepSeek V4.1 Flash high · EXL3 2.9bpw tie
    24%, 95% interval 10–47%
    =1Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt) tie
    24%, 95% interval 10–47%
    5MiniCPM-V 4.7 35B-A3B thinking on · BF16
    18%, 95% interval 6–41%
    =6Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt) tie
    12%, 95% interval 3–34%
    =6MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    12%, 95% interval 3–34%
    =6MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts tie
    12%, 95% interval 3–34%
    =9Gemma 4 31B thinking on · QAT W4A16 tie
    0%, 95% interval 0–18%
    =9Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) tie
    0%, 95% interval 0–18%
      Show the numbers
      ModelScore95% CI
      GLM-5.3 Flash low · EXL3 4.0bpw experts, FP8 dense24%10–47
      Qwen3.8 Flash-Next medium · NVFP424%10–47
      DeepSeek V4.1 Flash high · EXL3 2.9bpw24%10–47
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)24%10–47
      MiniCPM-V 4.7 35B-A3B thinking on · BF1618%6–41
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)12%3–34
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)12%3–34
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts12%3–34
      Gemma 4 31B thinking on · QAT W4A160%0–18
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)0%0–18
      Salesforce docs: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Claude Opus 5.5 effort high
      88%95% CI 66–97
      		17
      2Claude Opus 5.5 effort low
      82%95% CI 59–94
      		17
      2GPT-6 Astra effort medium
      82%95% CI 59–94
      		17
      4Claude Opus 5.5 effort medium
      76%95% CI 53–90
      		17
      4Claude Sonnet 5.5 effort high
      76%95% CI 53–90
      		17
      4Gemini 4 Argon effort medium
      76%95% CI 53–90
      		17
      4Gemini 4 Argon effort low
      76%95% CI 53–90
      		17
      4Gemini 4 Argon effort high
      76%95% CI 53–90
      		17
      4GPT-6.1 Sol effort high
      76%95% CI 53–90
      		17
      10GPT-6 Astra effort high
      65%95% CI 41–83
      		17
      10Grok 4.7 effort high
      65%95% CI 41–83
      		17
      12Claude Fable 5.1 effort high
      59%95% CI 36–78
      		17
      12Claude Opus 4.8 effort high
      59%95% CI 36–78
      		17
      12Claude Sonnet 5.5 effort low
      59%95% CI 36–78
      		17
      15Claude Opus 4.8 effort medium
      53%95% CI 31–74
      		17
      15GPT-5.6 Sol effort medium
      53%95% CI 31–74
      		17
      15Grok 4.6 effort high
      53%95% CI 31–74
      		17
      15Muse Wonder effort medium
      53%95% CI 31–74
      		17
      19Claude Opus 4.8 effort low
      47%95% CI 26–69
      		17
      19Claude Opus 4.8 thinking off
      47%95% CI 26–69
      		17
      19Gemini 3.8 Flash effort medium
      47%95% CI 26–69
      		17
      19GPT-6 Luna effort medium
      47%95% CI 26–69
      		17
      23Claude Sonnet 5.5 effort medium
      41%95% CI 22–64
      		17
      23Muse Wonder effort max
      41%95% CI 22–64
      		17
      25MiMo V2.6 Pro FP8 · thinking on
      35%95% CI 17–59
      		17
      25Muse Wonder effort low
      35%95% CI 17–59
      		17
      25Muse Wonder effort high
      35%95% CI 17–59
      		17
      28Claude Haiku 5.5 effort medium
      29%95% CI 13–53
      		17
      28DeepSeek V4.1 Flash FP8 · effort high
      29%95% CI 13–53
      		17
      30Claude Haiku 5.5 effort low
      24%95% CI 10–47
      		17
      30Claude Sonnet 5 effort high
      24%95% CI 10–47
      		17
      30Claude Sonnet 5 thinking off
      24%95% CI 10–47
      		17
      30Claude Sonnet 5 effort low
      24%95% CI 10–47
      		17
      30DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      24%95% CI 10–47
      		17
      30DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      24%95% CI 10–47
      		17
      30GLM-5.3 FP8 · effort low
      24%95% CI 10–47
      		17
      30GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      24%95% CI 10–47
      		17
      30GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      24%95% CI 10–47
      		17
      30Inkling Small FP8 · effort high
      24%95% CI 10–47
      		17
      30MiMo V2.6 Flash FP8 · thinking on
      24%95% CI 10–47
      		17
      30Nemotron 3 Ultra thinking on
      24%95% CI 10–47
      		17
      30Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      24%95% CI 10–47
      		17
      30Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      24%95% CI 10–47
      		17
      44Claude Haiku 5.5 effort high
      18%95% CI 6–41
      		17
      44Claude Sonnet 5 effort medium
      18%95% CI 6–41
      		17
      44DeepSeek V4 Pro FP8 · effort high
      18%95% CI 6–41
      		17
      44DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      18%95% CI 6–41
      		17
      44MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      18%95% CI 6–41
      		17
      44Mistral Large 4 effort high
      18%95% CI 6–41
      		17
      44Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      18%95% CI 6–41
      		17
      51Claude Haiku 4.5 thinking off
      12%95% CI 3–34
      		17
      51DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      12%95% CI 3–34
      		17
      51GLM-5.3 Flash FP8 · effort high
      12%95% CI 3–34
      		17
      51GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      12%95% CI 3–34
      		17
      51GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      12%95% CI 3–34
      		17
      51MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      12%95% CI 3–34
      		17
      51MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      12%95% CI 3–34
      		17
      51Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      12%95% CI 3–34
      		17
      51Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      12%95% CI 3–34
      		17
      51Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      12%95% CI 3–34
      		17
      51Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      12%95% CI 3–34
      		17
      62Laguna S 2.1 FP4 · thinking on
      6%95% CI 1–27
      		17
      62MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      6%95% CI 1–27
      		17
      62Qwen3.8 27B Splash 4-bit · Splash · effort low
      6%95% CI 1–27
      		17
      65Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      3%95% CI 0–23
      		17
      66Claude Haiku 4.5 thinking on
      0%95% CI 0–18
      		17
      66Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      0%95% CI 0–18
      		17
      66Gemma 4 31B QAT W4A16 · vLLM · thinking on
      0%95% CI 0–18
      		17
      66Ling 3.0 Flash BF16 · thinking on
      0%95% CI 0–18
      		17
      66Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      0%95% CI 0–18
      		17
      66Nemotron 3.5 Lightning NVFP4 · thinking on
      0%95% CI 0–18
      		17
      66Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      0%95% CI 0–18
      		17
      66Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      0%95% CI 0–18
      		17
      66Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      0%95% CI 0–18
      		17
      66Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      0%95% CI 0–18
      		17
      66Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      0%95% CI 0–18
      		17
      66Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      0%95% CI 0–18
      		17
      66Qwen3.8 Flash-Next thinking on
      0%95% CI 0–18
      		17

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      Salesforce docs tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      Concurrent long-running API request limitdocs-api-concurrent-long-runningeasymedium
      35%
      Default Apex callout timeoutdocs-callout-default-timeouteasyhard
      13%
      Maximum scratch org durationdocs-scratch-org-max-durationeasymedium
      53%
      REST response header with API usagedocs-rest-limit-info-headereasyhard
      19%
      Where to enable Lightning Web Securitydocs-lws-enable-settingeasyhard
      23%
      After trigger vs after-save flow orderdocs-order-exec-after-trigger-vs-flowmediummedium
      62%
      Batch Apex scope maximum with a QueryLocatordocs-batch-querylocator-scope-maxmediummedium
      61%
      Callout response size limit in async Apexdocs-callout-async-response-sizemediummedium
      49%
      Default platform event trigger batch sizedocs-pe-trigger-batch-sizemediumhard
      28%
      Jobs a running Queueable can enqueuedocs-queueable-async-enqueue-limitmediummedium
      51%
      Raw CSV size per Bulk API 2.0 ingest jobdocs-bulk2-ingest-upload-sizemediumhard
      14%
      What LWS uses instead of secure wrappersdocs-lws-distortionsmediumhard
      4%
      Compute an org's daily API request allocationdocs-api-daily-allocation-calchardhard
      15%
      Concurrent long-running Apex transactions for a large orgdocs-long-running-apex-concurrencyhardhard
      9%
      Cumulative cross-namespace SOQL limitdocs-cross-namespace-soql-limithardhard
      32%
      Platform event delivery allocation with several subscribersdocs-pe-delivery-allocation-calchardhard
      11%
      Text length budget in a big object indexdocs-big-object-index-text-lengthhardhard
      11%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au