Suites / scratch-def
Scratch org definitions
Turn a client brief into a working scratch org definition file (and sfdx-project.json). Answers are validated against the official JSON schemas, the documented scratch org feature list and the Metadata API settings types; settings are check-only deployed to a dedicated grader org exactly as scratch org creation applies them (types with lasting side effects, such as fiscal year or currencies, are validated offline). Task rules then check the brief's requirements (edition, features, settings values, locale).
How it's graded
Definition files are validated against the official JSON schemas and the documented scratch org feature list, and their settings are deployed check-only to a grader org exactly as scratch org creation applies them. Task rules then check the brief: edition, features, settings values and locale.
Difficulty mix
Leaderboard
Who's best at Scratch org definitions
pass@1 on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide, so read overlapping bars as ties. Open in the full leaderboard →
| # | Model and config | Score pass@1 · 95% CI | Tasks |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 17%95% CI 0–33 | 18 |
| 2 | Gemma 4 26B-A4B NVFP4 · vLLM · effort on | 11%95% CI 0–28 | 18 |
| 3 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 7%95% CI 0–20 | 18 |
| 4 | Gemma 4 31B QAT W4A16 · vLLM · effort on | 6%95% CI 0–17 | 18 |
| 4 | Qwen3.6 35B-A3B NVFP4 · vLLM · effort on | 6%95% CI 0–17 | 18 |
| 4 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 6%95% CI 0–17 | 18 |
| 4 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 6%95% CI 0–17 | 18 |
| 4 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 6%95% CI 0–17 | 18 |
| 4 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 6%95% CI 0–17 | 18 |
| 4 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 6%95% CI 0–17 | 18 |
| 4 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 6%95% CI 0–17 | 18 |
| 12 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 0%95% CI 0–0 | 18 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort highRun so far: 14 of 15 suites · 219 of 272 tasks graded | 21%95% CI 0–43 | 14 |
Tasks
What's in the suite
Solve rate is the mean pass@1 on that task across every finished run, a rough guide to how hard models find it.
| Task | Difficulty | Solve rate |
|---|---|---|
| B2C insurer with person accountsscratch-def-person-accounts | easy | 44% |
| Integration org with state and country picklistsscratch-def-state-country-picklists | easy | 0% |
| ISV regression org on the preview releasescratch-def-partner-preview-release | easy | 0% |
| Professional Edition org with API and Apexscratch-def-professional-edition-api | easy | 0% |
| sfdx-project.json for a sandbox-based teamscratch-def-sfdx-project-sandbox-team | easy | 50% |
| Australian locale and financial yearscratch-def-australian-locale | medium | 0% |
| Experience Cloud org for a guest-accessible sitescratch-def-experience-cloud-guest-site | medium | 0% |
| German subsidiary with multi-currencyscratch-def-multicurrency-germany | medium | 8% |
| Migrate a definition off orgPreferencesscratch-def-fix-org-preferences | medium | 0% |
| Scratch org for Agentforce and Prompt Builder workscratch-def-einstein-generative-ai | medium | 8% |
| Service Cloud console with Lightning Knowledge and Omni-Channelscratch-def-service-knowledge-omni | medium | 0% |
| Team selling with account and opportunity teamsscratch-def-opportunity-teams | medium | 8% |
| Field Service org with licences and multi-currencyscratch-def-field-service-multicurrency | hard | 0% |
| Repair a scratch org definition that fails creationscratch-def-fix-broken-definition | hard | 0% |
| Sales Cloud forecasting and pipeline orgscratch-def-sales-forecasting | hard | 0% |
| sfdx-project.json for a two-package 2GP monoreposcratch-def-sfdx-project-2gp-monorepo | hard | 0% |
| Sharing and record type prerequisites for a package installscratch-def-package-object-settings | hard | 0% |
| Shield and hardened security for a health clientscratch-def-shield-security | hard | 0% |
Work with Leo
Want this for your team?
Leo helps Salesforce teams run AI they own: open-weight models on infrastructure you control, tested on your kind of work before you rely on them. A private benchmark of your shortlist is a fixed fee.
Get a private benchmarkOr email leo@azl.au · azl.au