Suites / cli

Salesforce CLI

Day-to-day and CI/CD work with the Salesforce CLI (sf v2): deploy and retrieve by source, metadata and manifest, validation and quick deploy, destructive changes, Apex tests and anonymous Apex, scratch orgs and sandboxes, JWT auth, config and aliases, data exports and bulk loads, and unlocked packages. Answers are shell commands. Each command is parsed the way the CLI parses it and checked against the command manifest of a pinned @salesforce/cli release (unknown commands or flags, invalid option values, missing required flags and deprecated sfdx-style syntax fail), then compared with the expected commands and flag values.

20
tasks
Deterministic validators
grading
100%
highest c@3 · Claude Opus 4.8, joint first with 23 others
78
configurations with results

How it's graded

Each command is parsed the way the CLI parses it and checked against the command manifest of a pinned Salesforce CLI release: unknown commands or flags, invalid values, missing required flags and deprecated sfdx-style syntax fail. It is then compared with the expected commands and flag values.

Difficulty mix

6 easy8 medium6 hard

Leaderboard

Who's best at Salesforce CLI

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 20 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at Salesforce CLI
Within three attempts (c@3) on Salesforce CLI, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
=1Gemini 4 Argon low tie
100%, 95% interval 84–100%
=1GPT-6 Astra medium tie
100%, 95% interval 84–100%
=1Claude Opus 4.8 thinking off tie
100%, 95% interval 84–100%
=1Muse Wonder low tie
100%, 95% interval 84–100%
=1Grok 4.6 high tie
100%, 95% interval 84–100%
6DeepSeek V4.1 Flash high
95%, 95% interval 76–99%
7MiMo V2.6 Pro thinking on
90%, 95% interval 70–97%
8GLM-5.3 Flash high
75%, 95% interval 53–89%
9Mistral Large 4 high
70%, 95% interval 48–85%
10Inkling Small high
65%, 95% interval 43–82%
11Nemotron 3 Ultra thinking on
60%, 95% interval 39–78%
12Qwen3.8 Flash-Next thinking on
40%, 95% interval 22–61%
13Laguna S 2.1 thinking on
30%, 95% interval 15–52%
14Ling 3.0 Flash thinking on
15%, 95% interval 5–36%
    Show the numbers
    ModelScore95% CI
    Gemini 4 Argon low100%84–100
    GPT-6 Astra medium100%84–100
    Claude Opus 4.8 thinking off100%84–100
    Muse Wonder low100%84–100
    Grok 4.6 high100%84–100
    DeepSeek V4.1 Flash high95%76–99
    MiMo V2.6 Pro thinking on90%70–97
    GLM-5.3 Flash high75%53–89
    Mistral Large 4 high70%48–85
    Inkling Small high65%43–82
    Nemotron 3 Ultra thinking on60%39–78
    Qwen3.8 Flash-Next thinking on40%22–61
    Laguna S 2.1 thinking on30%15–52
    Ling 3.0 Flash thinking on15%5–36
    Local: the best of each lab at Salesforce CLI
    Within three attempts (c@3) on Salesforce CLI, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    1DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts
    95%, 95% interval 76–99%
    2GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense
    85%, 95% interval 64–95%
    3Qwen3.8 Flash-Next medium · NVFP4
    55%, 95% interval 34–74%
    4MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts
    45%, 95% interval 26–66%
    =5MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    40%, 95% interval 22–61%
    =5Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt) tie
    40%, 95% interval 22–61%
    7Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)
    35%, 95% interval 18–57%
    8Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)
    30%, 95% interval 15–52%
    9MiniCPM-V 4.7 35B-A3B thinking on · BF16
    20%, 95% interval 8–42%
    10Gemma 4 31B thinking on · QAT W4A16
    15%, 95% interval 5–36%
      Show the numbers
      ModelScore95% CI
      DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts95%76–99
      GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense85%64–95
      Qwen3.8 Flash-Next medium · NVFP455%34–74
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts45%26–66
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)40%22–61
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)40%22–61
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)35%18–57
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)30%15–52
      MiniCPM-V 4.7 35B-A3B thinking on · BF1620%8–42
      Gemma 4 31B thinking on · QAT W4A1615%5–36
      Salesforce CLI: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Claude Opus 4.8 effort high
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 effort medium
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 effort low
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 thinking off
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort high
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort medium
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort low
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort high
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort low
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort medium
      100%95% CI 84–100
      		20
      1Gemini 3.8 Flash effort medium
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort medium
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort low
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort high
      100%95% CI 84–100
      		20
      1GPT-5.6 Sol effort medium
      100%95% CI 84–100
      		20
      1GPT-6 Astra effort medium
      100%95% CI 84–100
      		20
      1GPT-6 Astra effort high
      100%95% CI 84–100
      		20
      1GPT-6 Luna effort medium
      100%95% CI 84–100
      		20
      1GPT-6.1 Sol effort high
      100%95% CI 84–100
      		20
      1Grok 4.6 effort high
      100%95% CI 84–100
      		20
      1Grok 4.7 effort high
      100%95% CI 84–100
      		20
      1Muse Wonder effort medium
      100%95% CI 84–100
      		20
      1Muse Wonder effort max
      100%95% CI 84–100
      		20
      1Muse Wonder effort low
      100%95% CI 84–100
      		20
      25Claude Haiku 5.5 effort high
      95%95% CI 76–99
      		20
      25Claude Haiku 5.5 effort low
      95%95% CI 76–99
      		20
      25DeepSeek V4.1 Flash FP8 · effort high
      95%95% CI 76–99
      		20
      25DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      95%95% CI 76–99
      		20
      25Muse Wonder effort high
      95%95% CI 76–99
      		20
      30Claude Sonnet 5 effort high
      90%95% CI 70–97
      		20
      30Claude Sonnet 5 effort medium
      90%95% CI 70–97
      		20
      30DeepSeek V4 Pro FP8 · effort high
      90%95% CI 70–97
      		20
      30MiMo V2.6 Pro FP8 · thinking on
      90%95% CI 70–97
      		20
      34Claude Fable 5.1 effort high
      85%95% CI 64–95
      		20
      34Claude Haiku 5.5 effort medium
      85%95% CI 64–95
      		20
      34Claude Sonnet 5 thinking off
      85%95% CI 64–95
      		20
      34Claude Sonnet 5 effort low
      85%95% CI 64–95
      		20
      34DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      85%95% CI 64–95
      		20
      34GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      85%95% CI 64–95
      		20
      40DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      80%95% CI 58–92
      		20
      41DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      75%95% CI 53–89
      		20
      41GLM-5.3 Flash FP8 · effort high
      75%95% CI 53–89
      		20
      43Mistral Large 4 effort high
      70%95% CI 48–85
      		20
      44Inkling Small FP8 · effort high
      65%95% CI 43–82
      		20
      45Claude Haiku 4.5 thinking on
      60%95% CI 39–78
      		20
      45GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      60%95% CI 39–78
      		20
      45GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      60%95% CI 39–78
      		20
      45Nemotron 3 Ultra thinking on
      60%95% CI 39–78
      		20
      49GLM-5.3 FP8 · effort low
      55%95% CI 34–74
      		20
      49MiMo V2.6 Flash FP8 · thinking on
      55%95% CI 34–74
      		20
      49Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      55%95% CI 34–74
      		20
      52Claude Haiku 4.5 thinking off
      50%95% CI 30–70
      		20
      52Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      50%95% CI 30–70
      		20
      52Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      50%95% CI 30–70
      		20
      55GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      45%95% CI 26–66
      		20
      55MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      45%95% CI 26–66
      		20
      55Qwen3.8 27B Splash 4-bit · Splash · effort low
      45%95% CI 26–66
      		20
      55Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      45%95% CI 26–66
      		20
      55Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      45%95% CI 26–66
      		20
      60MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      40%95% CI 22–61
      		20
      60Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      40%95% CI 22–61
      		20
      60Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      40%95% CI 22–61
      		20
      60Qwen3.8 Flash-Next thinking on
      40%95% CI 22–61
      		20
      64Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      35%95% CI 18–57
      		20
      65Laguna S 2.1 FP4 · thinking on
      30%95% CI 15–52
      		20
      65Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      30%95% CI 15–52
      		20
      65Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      30%95% CI 15–52
      		20
      65Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      30%95% CI 15–52
      		20
      69MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      20%95% CI 8–42
      		20
      70Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      15%95% CI 5–36
      		20
      70Gemma 4 31B QAT W4A16 · vLLM · thinking on
      15%95% CI 5–36
      		20
      70Ling 3.0 Flash BF16 · thinking on
      15%95% CI 5–36
      		20
      70MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      15%95% CI 5–36
      		20
      70Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      15%95% CI 5–36
      		20
      70Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      15%95% CI 5–36
      		20
      70Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      15%95% CI 5–36
      		20
      77Nemotron 3.5 Lightning NVFP4 · thinking on
      10%95% CI 3–30
      		20
      78Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      5%95% CI 1–24
      		20

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      Salesforce CLI tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      Assign a permission set to another usercli-assign-permset-usereasymedium
      53%
      Delete a scratch org without a promptcli-delete-scratch-orgeasyeasy
      77%
      Deploy a single Apex class to a sandboxcli-deploy-apex-classeasyeasy
      84%
      Execute anonymous Apex from a filecli-apex-run-script-fileeasyeasy
      86%
      Open an org at a Setup page in a given browsercli-org-open-setup-pageeasyeasy
      78%
      Stream debug logs with a specific debug levelcli-apex-tail-logeasymedium
      54%
      Bulk upsert a Windows CSV on an external IDcli-bulk-upsert-external-idmediummedium
      37%
      Create a scratch org from a definition filecli-scratch-org-createmediumeasy
      73%
      Export millions of records to CSVcli-bulk-export-csvmediumhard
      28%
      Generate a package.xml manifest from an orgcli-manifest-from-orgmediummedium
      38%
      Install a key-protected package version unattendedcli-package-install-cimediummedium
      61%
      Retrieve specific components by metadata type and namecli-retrieve-field-and-flexipagemediummedium
      64%
      Run Apex tests with coverage and JUnit output for CIcli-apex-tests-junit-coveragemediummedium
      52%
      Start a Developer Pro sandbox creation from a scriptcli-sandbox-create-asyncmediummedium
      37%
      CI job - JWT login to a sandbox, then deploycli-jwt-login-and-deployhardeasy
      70%
      Deploy and delete a still-referenced field in one deploymentcli-post-destructive-deployhardmedium
      56%
      First release of an unlocked package, end to endcli-unlocked-package-releasehardmedium
      49%
      Onboard a developer machine - login, aliases, local and global configcli-dev-machine-confighardmedium
      47%
      Seed a scratch org with related records via a data plancli-tree-export-import-seedhardmedium
      48%
      Validate a production release, then quick deploy itcli-validate-then-quick-deployhardmedium
      39%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au