Suites / scratch-def

Scratch org definitions

Turn a client brief into a working scratch org definition file (and sfdx-project.json). Answers are validated against the official JSON schemas, the documented scratch org feature list and the Metadata API settings types; settings are check-only deployed to a dedicated grader org exactly as scratch org creation applies them (types with lasting side effects, such as fiscal year or currencies, are validated offline). Task rules then check the brief's requirements (edition, features, settings values, locale).

18
tasks
Execution + validators
grading
94%
highest c@3 · Claude Opus 5.5, joint first with 1 other
78
configurations with results

How it's graded

Definition files are validated against the official JSON schemas and the documented scratch org feature list, and their settings are deployed check-only to a grader org exactly as scratch org creation applies them. Task rules then check the brief: edition, features, settings values and locale.

Difficulty mix

5 easy7 medium6 hard

Leaderboard

Who's best at Scratch org definitions

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at Scratch org definitions
Within three attempts (c@3) on Scratch org definitions, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
1Claude Opus 5.5 low
94%, 95% interval 74–99%
2Gemini 4 Argon low
89%, 95% interval 67–97%
3Grok 4.7 high
83%, 95% interval 61–94%
4GPT-5.6 Sol medium
78%, 95% interval 55–91%
5Muse Wonder low
67%, 95% interval 44–84%
6DeepSeek V4.1 Flash high
61%, 95% interval 39–80%
7GLM-5.3 Flash high
44%, 95% interval 25–66%
=8MiMo V2.6 Pro thinking on tie
39%, 95% interval 20–61%
=8Mistral Large 4 high tie
39%, 95% interval 20–61%
10Inkling Small high
33%, 95% interval 16–56%
11Nemotron 3 Ultra thinking on
28%, 95% interval 13–51%
=12Ling 3.0 Flash thinking on tie
11%, 95% interval 3–33%
=12Qwen3.8 Flash-Next thinking on tie
11%, 95% interval 3–33%
14Laguna S 2.1 thinking on
0%, 95% interval 0–18%
    Show the numbers
    ModelScore95% CI
    Claude Opus 5.5 low94%74–99
    Gemini 4 Argon low89%67–97
    Grok 4.7 high83%61–94
    GPT-5.6 Sol medium78%55–91
    Muse Wonder low67%44–84
    DeepSeek V4.1 Flash high61%39–80
    GLM-5.3 Flash high44%25–66
    MiMo V2.6 Pro thinking on39%20–61
    Mistral Large 4 high39%20–61
    Inkling Small high33%16–56
    Nemotron 3 Ultra thinking on28%13–51
    Ling 3.0 Flash thinking on11%3–33
    Qwen3.8 Flash-Next thinking on11%3–33
    Laguna S 2.1 thinking on0%0–18
    Local: the best of each lab at Scratch org definitions
    Within three attempts (c@3) on Scratch org definitions, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    1DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts
    56%, 95% interval 34–75%
    2GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense
    39%, 95% interval 20–61%
    3Qwen3.8 Flash-Next medium · NVFP4
    17%, 95% interval 6–39%
    =4MiniCPM-V 4.7 35B-A3B thinking on · BF16 tie
    11%, 95% interval 3–33%
    =4Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt) tie
    11%, 95% interval 3–33%
    =4Gemma 4 26B-A4B thinking on · NVFP4 tie
    11%, 95% interval 3–33%
    =4MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    11%, 95% interval 3–33%
    =4Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt) tie
    11%, 95% interval 3–33%
    =4Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) tie
    11%, 95% interval 3–33%
    =4MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts tie
    11%, 95% interval 3–33%
      Show the numbers
      ModelScore95% CI
      DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts56%34–75
      GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense39%20–61
      Qwen3.8 Flash-Next medium · NVFP417%6–39
      MiniCPM-V 4.7 35B-A3B thinking on · BF1611%3–33
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)11%3–33
      Gemma 4 26B-A4B thinking on · NVFP411%3–33
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)11%3–33
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)11%3–33
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)11%3–33
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts11%3–33
      Scratch org definitions: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Claude Opus 5.5 effort high
      94%95% CI 74–99
      		18
      1Claude Opus 5.5 effort low
      94%95% CI 74–99
      		18
      3Gemini 4 Argon effort medium
      89%95% CI 67–97
      		18
      3Gemini 4 Argon effort low
      89%95% CI 67–97
      		18
      3Gemini 4 Argon effort high
      89%95% CI 67–97
      		18
      6Claude Fable 5.1 effort high
      83%95% CI 61–94
      		18
      6Claude Opus 5.5 effort medium
      83%95% CI 61–94
      		18
      6Grok 4.7 effort high
      83%95% CI 61–94
      		18
      9Claude Sonnet 5.5 effort high
      78%95% CI 55–91
      		18
      9GPT-5.6 Sol effort medium
      78%95% CI 55–91
      		18
      11Claude Sonnet 5.5 effort low
      67%95% CI 44–84
      		18
      11GPT-6 Astra effort medium
      67%95% CI 44–84
      		18
      11Muse Wonder effort low
      67%95% CI 44–84
      		18
      11Muse Wonder effort high
      67%95% CI 44–84
      		18
      15DeepSeek V4.1 Flash FP8 · effort high
      61%95% CI 39–80
      		18
      15Gemini 3.8 Flash effort medium
      61%95% CI 39–80
      		18
      15GPT-6.1 Sol effort high
      61%95% CI 39–80
      		18
      15Grok 4.6 effort high
      61%95% CI 39–80
      		18
      15Muse Wonder effort medium
      61%95% CI 39–80
      		18
      15Muse Wonder effort max
      61%95% CI 39–80
      		18
      21Claude Opus 4.8 effort high
      56%95% CI 34–75
      		18
      21Claude Sonnet 5.5 effort medium
      56%95% CI 34–75
      		18
      21DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      56%95% CI 34–75
      		18
      21GPT-6 Luna effort medium
      56%95% CI 34–75
      		18
      25Claude Haiku 5.5 effort medium
      50%95% CI 29–71
      		18
      25Claude Opus 4.8 effort medium
      50%95% CI 29–71
      		18
      25Claude Sonnet 5 effort high
      50%95% CI 29–71
      		18
      25Claude Sonnet 5 effort medium
      50%95% CI 29–71
      		18
      25GPT-6 Astra effort high
      50%95% CI 29–71
      		18
      30Claude Opus 4.8 effort low
      44%95% CI 25–66
      		18
      30DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      44%95% CI 25–66
      		18
      30GLM-5.3 Flash FP8 · effort high
      44%95% CI 25–66
      		18
      33Claude Haiku 5.5 effort high
      39%95% CI 20–61
      		18
      33Claude Haiku 5.5 effort low
      39%95% CI 20–61
      		18
      33Claude Opus 4.8 thinking off
      39%95% CI 20–61
      		18
      33DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      39%95% CI 20–61
      		18
      33DeepSeek V4 Pro FP8 · effort high
      39%95% CI 20–61
      		18
      33GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      39%95% CI 20–61
      		18
      33MiMo V2.6 Pro FP8 · thinking on
      39%95% CI 20–61
      		18
      33Mistral Large 4 effort high
      39%95% CI 20–61
      		18
      41Claude Sonnet 5 thinking off
      33%95% CI 16–56
      		18
      41Inkling Small FP8 · effort high
      33%95% CI 16–56
      		18
      43Claude Sonnet 5 effort low
      28%95% CI 13–51
      		18
      43GLM-5.3 FP8 · effort low
      28%95% CI 13–51
      		18
      43Nemotron 3 Ultra thinking on
      28%95% CI 13–51
      		18
      46GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      22%95% CI 9–45
      		18
      46GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      22%95% CI 9–45
      		18
      48DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      17%95% CI 6–39
      		18
      48Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      17%95% CI 6–39
      		18
      48Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      17%95% CI 6–39
      		18
      51Claude Haiku 4.5 thinking on
      11%95% CI 3–33
      		18
      51Claude Haiku 4.5 thinking off
      11%95% CI 3–33
      		18
      51Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      11%95% CI 3–33
      		18
      51Ling 3.0 Flash BF16 · thinking on
      11%95% CI 3–33
      		18
      51MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      11%95% CI 3–33
      		18
      51MiMo V2.6 Flash FP8 · thinking on
      11%95% CI 3–33
      		18
      51MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      11%95% CI 3–33
      		18
      51MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      11%95% CI 3–33
      		18
      51MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      11%95% CI 3–33
      		18
      51Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      11%95% CI 3–33
      		18
      51Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      11%95% CI 3–33
      		18
      51Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      11%95% CI 3–33
      		18
      51Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      11%95% CI 3–33
      		18
      51Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      11%95% CI 3–33
      		18
      51Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      11%95% CI 3–33
      		18
      51Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      11%95% CI 3–33
      		18
      51Qwen3.8 Flash-Next thinking on
      11%95% CI 3–33
      		18
      68Gemma 4 31B QAT W4A16 · vLLM · thinking on
      6%95% CI 1–26
      		18
      68GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      6%95% CI 1–26
      		18
      68Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      6%95% CI 1–26
      		18
      68Nemotron 3.5 Lightning NVFP4 · thinking on
      6%95% CI 1–26
      		18
      68Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      6%95% CI 1–26
      		18
      68Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      6%95% CI 1–26
      		18
      68Qwen3.8 27B Splash 4-bit · Splash · effort low
      6%95% CI 1–26
      		18
      68Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      6%95% CI 1–26
      		18
      68Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      6%95% CI 1–26
      		18
      68Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      6%95% CI 1–26
      		18
      78Laguna S 2.1 FP4 · thinking on
      0%95% CI 0–18
      		18

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      Scratch org definitions tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      B2C insurer with person accountsscratch-def-person-accountseasyeasy
      71%
      Integration org with state and country picklistsscratch-def-state-country-picklistseasyhard
      30%
      ISV regression org on the preview releasescratch-def-partner-preview-releaseeasyhard
      16%
      Professional Edition org with API and Apexscratch-def-professional-edition-apieasyhard
      20%
      sfdx-project.json for a sandbox-based teamscratch-def-sfdx-project-sandbox-teameasyeasy
      67%
      Australian locale and financial yearscratch-def-australian-localemediumhard
      4%
      Experience Cloud org for a guest-accessible sitescratch-def-experience-cloud-guest-sitemediumhard
      6%
      German subsidiary with multi-currencyscratch-def-multicurrency-germanymediumhard
      1%
      Migrate a definition off orgPreferencesscratch-def-fix-org-preferencesmediumhard
      27%
      Scratch org for Agentforce and Prompt Builder workscratch-def-einstein-generative-aimediumhard
      16%
      Service Cloud console with Lightning Knowledge and Omni-Channelscratch-def-service-knowledge-omnimediumhard
      11%
      Team selling with account and opportunity teamsscratch-def-opportunity-teamsmediumhard
      28%
      Field Service org with licences and multi-currencyscratch-def-field-service-multicurrencyhardhard
      0%
      Repair a scratch org definition that fails creationscratch-def-fix-broken-definitionhardhard
      0%
      Sales Cloud forecasting and pipeline orgscratch-def-sales-forecastinghardhard
      0%
      sfdx-project.json for a two-package 2GP monoreposcratch-def-sfdx-project-2gp-monorepohardhard
      27%
      Sharing and record type prerequisites for a package installscratch-def-package-object-settingshardhard
      16%
      Shield and hardened security for a health clientscratch-def-shield-securityhardhard
      8%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au