Suites / packaging

Packaging

Package development with unlocked, second-generation (2GP) and first-generation (1GP) managed packages: sfdx-project.json package directories, aliases, dependencies, ancestry and test setup; `sf package` and `sf package1` commands for creating, versioning, promoting, installing, upgrading and push-upgrading packages; and the platform rules behind them (IDs, promotion requirements, upgrade paths, component removal). Graded deterministically: project files are resolved the way the CLI resolves them, commands are validated against the pinned `sf` command manifest, knowledge questions are multiple choice.

18
tasks
Deterministic validators
grading
100%
highest c@3 · GPT-6 Astra
78
configurations with results

How it's graded

sfdx-project.json files are resolved the way the CLI resolves them, sf package and sf package1 commands are validated against the pinned sf command manifest, and packaging-judgement questions (1GP, managed 2GP or unlocked) are multiple choice.

Difficulty mix

5 easy7 medium6 hard

Leaderboard

Who's best at Packaging

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at Packaging
Within three attempts (c@3) on Packaging, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
1GPT-6 Astra medium
100%, 95% interval 82–100%
=2Gemini 4 Argon medium tie
94%, 95% interval 74–99%
=2Claude Opus 5.5 low tie
94%, 95% interval 74–99%
4Muse Wonder low
89%, 95% interval 67–97%
5Grok 4.6 high
83%, 95% interval 61–94%
6DeepSeek V4.1 Flash high
78%, 95% interval 55–91%
7MiMo V2.6 Pro thinking on
61%, 95% interval 39–80%
8Inkling Small high
56%, 95% interval 34–75%
9GLM-5.3 low
50%, 95% interval 29–71%
10Qwen3.8 Flash-Next thinking on
44%, 95% interval 25–66%
=11Nemotron 3 Ultra thinking on tie
39%, 95% interval 20–61%
=11Mistral Large 4 high tie
39%, 95% interval 20–61%
13Laguna S 2.1 thinking on
22%, 95% interval 9–45%
14Ling 3.0 Flash thinking on
17%, 95% interval 6–39%
    Show the numbers
    ModelScore95% CI
    GPT-6 Astra medium100%82–100
    Gemini 4 Argon medium94%74–99
    Claude Opus 5.5 low94%74–99
    Muse Wonder low89%67–97
    Grok 4.6 high83%61–94
    DeepSeek V4.1 Flash high78%55–91
    MiMo V2.6 Pro thinking on61%39–80
    Inkling Small high56%34–75
    GLM-5.3 low50%29–71
    Qwen3.8 Flash-Next thinking on44%25–66
    Nemotron 3 Ultra thinking on39%20–61
    Mistral Large 4 high39%20–61
    Laguna S 2.1 thinking on22%9–45
    Ling 3.0 Flash thinking on17%6–39
    Local: the best of each lab at Packaging
    Within three attempts (c@3) on Packaging, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    1DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts
    67%, 95% interval 44–84%
    =2GLM-5.3 Flash low · EXL3 4.0bpw experts, FP8 dense tie
    50%, 95% interval 29–71%
    =2Qwen3.8 Flash-Next medium · INT4-AutoRound, 5 of 10 experts tie
    50%, 95% interval 29–71%
    4MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts
    39%, 95% interval 20–61%
    5Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)
    33%, 95% interval 16–56%
    =6MiniCPM-V 4.7 35B-A3B thinking on · INT4 W4A16 experts, BF16 dense tie
    28%, 95% interval 13–51%
    =6MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    28%, 95% interval 13–51%
    =6Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) tie
    28%, 95% interval 13–51%
    9Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)
    22%, 95% interval 9–45%
    10Gemma 4 31B thinking on · QAT W4A16
    17%, 95% interval 6–39%
      Show the numbers
      ModelScore95% CI
      DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts67%44–84
      GLM-5.3 Flash low · EXL3 4.0bpw experts, FP8 dense50%29–71
      Qwen3.8 Flash-Next medium · INT4-AutoRound, 5 of 10 experts50%29–71
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts39%20–61
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)33%16–56
      MiniCPM-V 4.7 35B-A3B thinking on · INT4 W4A16 experts, BF16 dense28%13–51
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)28%13–51
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)28%13–51
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)22%9–45
      Gemma 4 31B thinking on · QAT W4A1617%6–39
      Packaging: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1GPT-6 Astra effort medium
      100%95% CI 82–100
      		18
      2Claude Opus 5.5 effort high
      94%95% CI 74–99
      		18
      2Claude Opus 5.5 effort medium
      94%95% CI 74–99
      		18
      2Claude Opus 5.5 effort low
      94%95% CI 74–99
      		18
      2Gemini 4 Argon effort medium
      94%95% CI 74–99
      		18
      2GPT-6 Astra effort high
      94%95% CI 74–99
      		18
      2GPT-6.1 Sol effort high
      94%95% CI 74–99
      		18
      8Claude Opus 4.8 effort high
      89%95% CI 67–97
      		18
      8Gemini 4 Argon effort low
      89%95% CI 67–97
      		18
      8Gemini 4 Argon effort high
      89%95% CI 67–97
      		18
      8GPT-5.6 Sol effort medium
      89%95% CI 67–97
      		18
      8Muse Wonder effort medium
      89%95% CI 67–97
      		18
      8Muse Wonder effort low
      89%95% CI 67–97
      		18
      14Claude Fable 5.1 effort high
      83%95% CI 61–94
      		18
      14Claude Opus 4.8 effort medium
      83%95% CI 61–94
      		18
      14Claude Sonnet 5.5 effort high
      83%95% CI 61–94
      		18
      14Claude Sonnet 5.5 effort low
      83%95% CI 61–94
      		18
      14Grok 4.6 effort high
      83%95% CI 61–94
      		18
      14Grok 4.7 effort high
      83%95% CI 61–94
      		18
      14Muse Wonder effort high
      83%95% CI 61–94
      		18
      21Claude Opus 4.8 effort low
      78%95% CI 55–91
      		18
      21Claude Sonnet 5.5 effort medium
      78%95% CI 55–91
      		18
      21DeepSeek V4.1 Flash FP8 · effort high
      78%95% CI 55–91
      		18
      21Gemini 3.8 Flash effort medium
      78%95% CI 55–91
      		18
      21GPT-6 Luna effort medium
      78%95% CI 55–91
      		18
      21Muse Wonder effort max
      78%95% CI 55–91
      		18
      27Claude Opus 4.8 thinking off
      72%95% CI 49–88
      		18
      27Claude Sonnet 5 effort high
      72%95% CI 49–88
      		18
      29Claude Haiku 5.5 effort medium
      67%95% CI 44–84
      		18
      29Claude Sonnet 5 effort medium
      67%95% CI 44–84
      		18
      29DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      67%95% CI 44–84
      		18
      32Claude Haiku 5.5 effort high
      61%95% CI 39–80
      		18
      32Claude Haiku 5.5 effort low
      61%95% CI 39–80
      		18
      32MiMo V2.6 Pro FP8 · thinking on
      61%95% CI 39–80
      		18
      35Claude Sonnet 5 effort low
      56%95% CI 34–75
      		18
      35Inkling Small FP8 · effort high
      56%95% CI 34–75
      		18
      37Claude Sonnet 5 thinking off
      50%95% CI 29–71
      		18
      37DeepSeek V4 Pro FP8 · effort high
      50%95% CI 29–71
      		18
      37GLM-5.3 FP8 · effort low
      50%95% CI 29–71
      		18
      37GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      50%95% CI 29–71
      		18
      37Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      50%95% CI 29–71
      		18
      42DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      44%95% CI 25–66
      		18
      42DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      44%95% CI 25–66
      		18
      42GLM-5.3 Flash FP8 · effort high
      44%95% CI 25–66
      		18
      42MiMo V2.6 Flash FP8 · thinking on
      44%95% CI 25–66
      		18
      42Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      44%95% CI 25–66
      		18
      42Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      44%95% CI 25–66
      		18
      42Qwen3.8 Flash-Next thinking on
      44%95% CI 25–66
      		18
      49DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      39%95% CI 20–61
      		18
      49GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      39%95% CI 20–61
      		18
      49GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      39%95% CI 20–61
      		18
      49MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      39%95% CI 20–61
      		18
      49Mistral Large 4 effort high
      39%95% CI 20–61
      		18
      49Nemotron 3 Ultra thinking on
      39%95% CI 20–61
      		18
      49Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      39%95% CI 20–61
      		18
      49Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      39%95% CI 20–61
      		18
      57Claude Haiku 4.5 thinking on
      33%95% CI 16–56
      		18
      57GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      33%95% CI 16–56
      		18
      57Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      33%95% CI 16–56
      		18
      60Claude Haiku 4.5 thinking off
      28%95% CI 13–51
      		18
      60MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      28%95% CI 13–51
      		18
      60MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      28%95% CI 13–51
      		18
      60Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      28%95% CI 13–51
      		18
      60Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      28%95% CI 13–51
      		18
      60Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      28%95% CI 13–51
      		18
      66Laguna S 2.1 FP4 · thinking on
      22%95% CI 9–45
      		18
      66Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      22%95% CI 9–45
      		18
      66Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      22%95% CI 9–45
      		18
      66Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      22%95% CI 9–45
      		18
      70Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      19%95% CI 7–42
      		18
      71Gemma 4 31B QAT W4A16 · vLLM · thinking on
      17%95% CI 6–39
      		18
      71Ling 3.0 Flash BF16 · thinking on
      17%95% CI 6–39
      		18
      71MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      17%95% CI 6–39
      		18
      71Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      17%95% CI 6–39
      		18
      71Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      17%95% CI 6–39
      		18
      71Qwen3.8 27B Splash 4-bit · Splash · effort low
      17%95% CI 6–39
      		18
      77Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      11%95% CI 3–33
      		18
      77Nemotron 3.5 Lightning NVFP4 · thinking on
      11%95% CI 3–33
      		18

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      Packaging tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      Configure a new unlocked package in sfdx-project.jsonpackaging-unlocked-project-configeasymedium
      65%
      How second-generation managed packaging differs from 1GPpackaging-1gp-vs-2gpeasyeasy
      99%
      Match packaging IDs to the commands that use thempackaging-package-id-prefixeseasyeasy
      83%
      Package metadata that depends on unpackaged org metadatapackaging-create-org-dependenteasyhard
      33%
      Stop installs of an accidentally promoted managed versionpackaging-lock-promoted-versioneasymedium
      53%
      Build a promotable package version in CIpackaging-version-create-release-candidatemediummedium
      44%
      Build an extension package against the promoted base versionpackaging-dependency-released-buildmediumeasy
      81%
      Create an unlocked package without the project namespacepackaging-create-unlocked-no-namespacemediummedium
      47%
      Managed 2GP project with ancestry and test-only metadatapackaging-managed-test-setupmediumhard
      29%
      Promote a package version and install it in production from CIpackaging-promote-and-installmediummedium
      35%
      Schedule a push upgrade for a wave of subscriber orgspackaging-push-upgrade-schedulemediumhard
      11%
      What --skip-validation means for a release pipelinepackaging-skip-validation-consequencesmediummedium
      58%
      Abandon a released managed 2GP version through ancestrypackaging-abandon-ancestorhardmedium
      56%
      Declare multi-level dependencies across unlocked packagespackaging-transitive-dependencieshardhard
      22%
      Move metadata from one unlocked package to another in productionpackaging-migrate-metadata-between-packageshardhard
      27%
      Upload a Managed - Released first-generation package versionpackaging-1gp-managed-released-uploadhardhard
      33%
      Which removed managed 2GP components are hard deleted on upgrade?packaging-removed-components-upgradehardhard
      4%
      Which subscriber upgrades does a 2GP ancestry tree allow?packaging-ancestry-upgrade-pathshardhard
      25%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au