Suites / cli
Salesforce CLI
Day-to-day and CI/CD work with the Salesforce CLI (sf v2): deploy and retrieve by source, metadata and manifest, validation and quick deploy, destructive changes, Apex tests and anonymous Apex, scratch orgs and sandboxes, JWT auth, config and aliases, data exports and bulk loads, and unlocked packages. Answers are shell commands. Each command is parsed the way the CLI parses it and checked against the command manifest of a pinned @salesforce/cli release (unknown commands or flags, invalid option values, missing required flags and deprecated sfdx-style syntax fail), then compared with the expected commands and flag values.
How it's graded
Each command is parsed the way the CLI parses it and checked against the command manifest of a pinned Salesforce CLI release: unknown commands or flags, invalid values, missing required flags and deprecated sfdx-style syntax fail. It is then compared with the expected commands and flag values.
Difficulty mix
Leaderboard
Who's best at Salesforce CLI
pass@1 on this suite only, with 95% confidence intervals. With 20 tasks the intervals are wide, so read overlapping bars as ties. Open in the full leaderboard →
| # | Model and config | Score pass@1 · 95% CI | Tasks |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 85%95% CI 70–100 | 20 |
| 2 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 35%95% CI 15–55 | 20 |
| 2 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 35%95% CI 15–55 | 20 |
| 4 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 30%95% CI 10–50 | 20 |
| 5 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 25%95% CI 10–45 | 20 |
| 5 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 25%95% CI 10–45 | 20 |
| 7 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 20%95% CI 5–40 | 20 |
| 7 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 20%95% CI 5–40 | 20 |
| 9 | Gemma 4 26B-A4B NVFP4 · vLLM · effort on | 15%95% CI 0–30 | 20 |
| 9 | Gemma 4 31B QAT W4A16 · vLLM · effort on | 15%95% CI 0–30 | 20 |
| 9 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 15%95% CI 3–30 | 20 |
| 12 | Qwen3.6 35B-A3B NVFP4 · vLLM · effort on | 10%95% CI 0–25 | 20 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort highRun so far: 14 of 15 suites · 219 of 272 tasks graded | 45%95% CI 25–65 | 20 |
Tasks
What's in the suite
Solve rate is the mean pass@1 on that task across every finished run, a rough guide to how hard models find it.
| Task | Difficulty | Solve rate |
|---|---|---|
| Assign a permission set to another usercli-assign-permset-user | easy | 8% |
| Delete a scratch org without a promptcli-delete-scratch-org | easy | 67% |
| Deploy a single Apex class to a sandboxcli-deploy-apex-class | easy | 89% |
| Execute anonymous Apex from a filecli-apex-run-script-file | easy | 83% |
| Open an org at a Setup page in a given browsercli-org-open-setup-page | easy | 75% |
| Stream debug logs with a specific debug levelcli-apex-tail-log | easy | 8% |
| Bulk upsert a Windows CSV on an external IDcli-bulk-upsert-external-id | medium | 0% |
| Create a scratch org from a definition filecli-scratch-org-create | medium | 50% |
| Export millions of records to CSVcli-bulk-export-csv | medium | 8% |
| Generate a package.xml manifest from an orgcli-manifest-from-org | medium | 0% |
| Install a key-protected package version unattendedcli-package-install-ci | medium | 17% |
| Retrieve specific components by metadata type and namecli-retrieve-field-and-flexipage | medium | 44% |
| Run Apex tests with coverage and JUnit output for CIcli-apex-tests-junit-coverage | medium | 25% |
| Start a Developer Pro sandbox creation from a scriptcli-sandbox-create-async | medium | 0% |
| CI job - JWT login to a sandbox, then deploycli-jwt-login-and-deploy | hard | 33% |
| Deploy and delete a still-referenced field in one deploymentcli-post-destructive-deploy | hard | 8% |
| First release of an unlocked package, end to endcli-unlocked-package-release | hard | 8% |
| Onboard a developer machine - login, aliases, local and global configcli-dev-machine-config | hard | 8% |
| Seed a scratch org with related records via a data plancli-tree-export-import-seed | hard | 8% |
| Validate a production release, then quick deploy itcli-validate-then-quick-deploy | hard | 8% |
Work with Leo
Want this for your team?
Leo helps Salesforce teams run AI they own: open-weight models on infrastructure you control, tested on your kind of work before you rely on them. A private benchmark of your shortlist is a fixed fee.
Get a private benchmarkOr email leo@azl.au · azl.au