Suites / ci

CI/CD

Salesforce CI/CD engineering: GitHub Actions workflows and pipeline scripts that authenticate with JWT or SFDX auth URLs from secrets, validate pull requests, quick deploy validated releases, run delta deployments with sfdx-git-delta, manage scratch orgs per pull request, build and promote unlocked packages, gate builds on Code Analyzer and run LWC Jest, plus CI/CD judgement questions. Every sf command in an answer is validated against the pinned sf CLI manifest; workflow properties (triggers, job ordering, environments, cleanup, secrets) are checked deterministically, including simulated events. No org is needed.

17
tasks
Deterministic validators
grading
100%
highest c@3 · Claude Opus 4.8, joint first with 6 others
78
configurations with results

How it's graded

GitHub Actions workflows are checked for structure and secrets hygiene, and every sf command in them is validated against the pinned sf command manifest.

Difficulty mix

5 easy7 medium5 hard

Leaderboard

Who's best at CI/CD

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 17 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at CI/CD
Within three attempts (c@3) on CI/CD, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
=1Gemini 4 Argon low tie
100%, 95% interval 82–100%
=1GPT-6 Astra medium tie
100%, 95% interval 82–100%
=1Claude Opus 4.8 thinking off tie
100%, 95% interval 82–100%
4Grok 4.6 high
94%, 95% interval 73–99%
=5GLM-5.3 low tie
88%, 95% interval 66–97%
=5Muse Wonder low tie
88%, 95% interval 66–97%
=5DeepSeek V4.1 Flash high tie
88%, 95% interval 66–97%
8Mistral Large 4 high
82%, 95% interval 59–94%
=9Nemotron 3 Ultra thinking on tie
71%, 95% interval 47–87%
=9MiMo V2.6 Pro thinking on tie
71%, 95% interval 47–87%
11Inkling Small high
65%, 95% interval 41–83%
12Qwen3.8 Flash-Next thinking on
59%, 95% interval 36–78%
13Laguna S 2.1 thinking on
41%, 95% interval 22–64%
14Ling 3.0 Flash thinking on
35%, 95% interval 17–59%
    Show the numbers
    ModelScore95% CI
    Gemini 4 Argon low100%82–100
    GPT-6 Astra medium100%82–100
    Claude Opus 4.8 thinking off100%82–100
    Grok 4.6 high94%73–99
    GLM-5.3 low88%66–97
    Muse Wonder low88%66–97
    DeepSeek V4.1 Flash high88%66–97
    Mistral Large 4 high82%59–94
    Nemotron 3 Ultra thinking on71%47–87
    MiMo V2.6 Pro thinking on71%47–87
    Inkling Small high65%41–83
    Qwen3.8 Flash-Next thinking on59%36–78
    Laguna S 2.1 thinking on41%22–64
    Ling 3.0 Flash thinking on35%17–59
    Local: the best of each lab at CI/CD
    Within three attempts (c@3) on CI/CD, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    =1GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense tie
    82%, 95% interval 59–94%
    =1DeepSeek V4.1 Flash high · EXL3 2.9bpw tie
    82%, 95% interval 59–94%
    3Qwen3.8 Flash-Next medium · NVFP4
    71%, 95% interval 47–87%
    4Gemma 4 31B thinking on · QAT W4A16
    59%, 95% interval 36–78%
    =5Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt) tie
    53%, 95% interval 31–74%
    =5Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt) tie
    53%, 95% interval 31–74%
    =7MiniCPM-V 4.7 35B-A3B thinking on · BF16 tie
    35%, 95% interval 17–59%
    =7MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    35%, 95% interval 17–59%
    =7Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) tie
    35%, 95% interval 17–59%
    =7MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts tie
    35%, 95% interval 17–59%
      Show the numbers
      ModelScore95% CI
      GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense82%59–94
      DeepSeek V4.1 Flash high · EXL3 2.9bpw82%59–94
      Qwen3.8 Flash-Next medium · NVFP471%47–87
      Gemma 4 31B thinking on · QAT W4A1659%36–78
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)53%31–74
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)53%31–74
      MiniCPM-V 4.7 35B-A3B thinking on · BF1635%17–59
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)35%17–59
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)35%17–59
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts35%17–59
      CI/CD: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Claude Opus 4.8 effort medium
      100%95% CI 82–100
      		17
      1Claude Opus 4.8 effort low
      100%95% CI 82–100
      		17
      1Claude Opus 4.8 thinking off
      100%95% CI 82–100
      		17
      1Claude Sonnet 5.5 effort high
      100%95% CI 82–100
      		17
      1Gemini 4 Argon effort medium
      100%95% CI 82–100
      		17
      1Gemini 4 Argon effort low
      100%95% CI 82–100
      		17
      1GPT-6 Astra effort medium
      100%95% CI 82–100
      		17
      8Claude Fable 5.1 effort high
      94%95% CI 73–99
      		17
      8Claude Haiku 5.5 effort medium
      94%95% CI 73–99
      		17
      8Claude Opus 4.8 effort high
      94%95% CI 73–99
      		17
      8Claude Opus 5.5 effort high
      94%95% CI 73–99
      		17
      8Claude Opus 5.5 effort low
      94%95% CI 73–99
      		17
      8Claude Sonnet 5 effort medium
      94%95% CI 73–99
      		17
      8GPT-6 Astra effort high
      94%95% CI 73–99
      		17
      8GPT-6.1 Sol effort high
      94%95% CI 73–99
      		17
      8Grok 4.6 effort high
      94%95% CI 73–99
      		17
      17Claude Haiku 5.5 effort high
      88%95% CI 66–97
      		17
      17Claude Opus 5.5 effort medium
      88%95% CI 66–97
      		17
      17Claude Sonnet 5 thinking off
      88%95% CI 66–97
      		17
      17Claude Sonnet 5.5 effort low
      88%95% CI 66–97
      		17
      17Claude Sonnet 5.5 effort medium
      88%95% CI 66–97
      		17
      17DeepSeek V4 Pro FP8 · effort high
      88%95% CI 66–97
      		17
      17DeepSeek V4.1 Flash FP8 · effort high
      88%95% CI 66–97
      		17
      17Gemini 3.8 Flash effort medium
      88%95% CI 66–97
      		17
      17Gemini 4 Argon effort high
      88%95% CI 66–97
      		17
      17GLM-5.3 FP8 · effort low
      88%95% CI 66–97
      		17
      17GPT-5.6 Sol effort medium
      88%95% CI 66–97
      		17
      17GPT-6 Luna effort medium
      88%95% CI 66–97
      		17
      17Grok 4.7 effort high
      88%95% CI 66–97
      		17
      17Muse Wonder effort max
      88%95% CI 66–97
      		17
      17Muse Wonder effort low
      88%95% CI 66–97
      		17
      17Muse Wonder effort high
      88%95% CI 66–97
      		17
      33Claude Haiku 5.5 effort low
      82%95% CI 59–94
      		17
      33DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      82%95% CI 59–94
      		17
      33DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      82%95% CI 59–94
      		17
      33GLM-5.3 Flash FP8 · effort high
      82%95% CI 59–94
      		17
      33GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      82%95% CI 59–94
      		17
      33Mistral Large 4 effort high
      82%95% CI 59–94
      		17
      33Muse Wonder effort medium
      82%95% CI 59–94
      		17
      40Claude Sonnet 5 effort high
      76%95% CI 53–90
      		17
      40DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      76%95% CI 53–90
      		17
      40GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      76%95% CI 53–90
      		17
      43Claude Sonnet 5 effort low
      71%95% CI 47–87
      		17
      43MiMo V2.6 Pro FP8 · thinking on
      71%95% CI 47–87
      		17
      43Nemotron 3 Ultra thinking on
      71%95% CI 47–87
      		17
      43Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      71%95% CI 47–87
      		17
      47Claude Haiku 4.5 thinking on
      65%95% CI 41–83
      		17
      47GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      65%95% CI 41–83
      		17
      47Inkling Small FP8 · effort high
      65%95% CI 41–83
      		17
      47MiMo V2.6 Flash FP8 · thinking on
      65%95% CI 41–83
      		17
      47Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      65%95% CI 41–83
      		17
      52Claude Haiku 4.5 thinking off
      59%95% CI 36–78
      		17
      52Gemma 4 31B QAT W4A16 · vLLM · thinking on
      59%95% CI 36–78
      		17
      52Qwen3.8 Flash-Next thinking on
      59%95% CI 36–78
      		17
      55DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      53%95% CI 31–74
      		17
      55Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      53%95% CI 31–74
      		17
      55Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      53%95% CI 31–74
      		17
      55Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      53%95% CI 31–74
      		17
      55Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      53%95% CI 31–74
      		17
      60GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      47%95% CI 26–69
      		17
      60Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      47%95% CI 26–69
      		17
      62Laguna S 2.1 FP4 · thinking on
      41%95% CI 22–64
      		17
      62Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      41%95% CI 22–64
      		17
      62Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      41%95% CI 22–64
      		17
      62Qwen3.8 27B Splash 4-bit · Splash · effort low
      41%95% CI 22–64
      		17
      62Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      41%95% CI 22–64
      		17
      67Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      35%95% CI 17–59
      		17
      67Ling 3.0 Flash BF16 · thinking on
      35%95% CI 17–59
      		17
      67MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      35%95% CI 17–59
      		17
      67MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      35%95% CI 17–59
      		17
      67MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      35%95% CI 17–59
      		17
      67MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      35%95% CI 17–59
      		17
      67Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      35%95% CI 17–59
      		17
      67Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      35%95% CI 17–59
      		17
      75Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      29%95% CI 13–53
      		17
      75Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      29%95% CI 13–53
      		17
      75Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      29%95% CI 13–53
      		17
      78Nemotron 3.5 Lightning NVFP4 · thinking on
      18%95% CI 6–41
      		17

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      CI/CD tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      Deploy to a UAT sandbox with an SFDX auth URLci-uat-deploy-sfdx-urleasymedium
      52%
      LWC lint and Jest job with cached npm dependenciesci-lwc-jest-eslinteasyeasy
      100%
      Nightly Apex regression run in a full sandboxci-nightly-apex-testseasymedium
      34%
      Pre or post destructive changes for a referenced fieldci-choice-destructive-ordereasyeasy
      95%
      Test level for a production deploy with managed packagesci-choice-test-level-prodeasyeasy
      99%
      Block pull requests on high-severity Code Analyzer violationsci-code-analyzer-gatemediummedium
      37%
      Fix a production deploy that lost its environment secretsci-environment-secrets-fixmediumeasy
      99%
      Scratch orgs versus sandboxes as per-pull-request CI orgsci-choice-scratch-vs-sandboxmediumeasy
      95%
      Script a disposable review scratch orgci-review-org-scriptmediummedium
      51%
      Source tracking behaviour in CI jobsci-choice-source-tracking-cimediummedium
      65%
      Validate pull requests against production with JWT authci-pr-validate-jwtmediummedium
      34%
      Why a quick deploy finds no recent job IDci-choice-quick-deploy-cachemediumeasy
      99%
      Build and promote an unlocked package version on release tagsci-unlocked-package-releasehardhard
      28%
      Delta validation with sfdx-git-delta and destructive changesci-delta-deploy-sgdhardhard
      18%
      One workflow that routes validations and deploys by branchci-branch-routed-validationhardmedium
      47%
      Throwaway scratch org per pull request with guaranteed cleanupci-scratch-org-per-prhardmedium
      43%
      Validate on merge, then quick deploy after approvalci-quick-deploy-on-mergehardmedium
      39%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au