Suites / packaging
Packaging
Package development with unlocked, second-generation (2GP) and first-generation (1GP) managed packages: sfdx-project.json package directories, aliases, dependencies, ancestry and test setup; `sf package` and `sf package1` commands for creating, versioning, promoting, installing, upgrading and push-upgrading packages; and the platform rules behind them (IDs, promotion requirements, upgrade paths, component removal). Graded deterministically: project files are resolved the way the CLI resolves them, commands are validated against the pinned `sf` command manifest, knowledge questions are multiple choice.
How it's graded
sfdx-project.json files are resolved the way the CLI resolves them, sf package and sf package1 commands are validated against the pinned sf command manifest, and packaging-judgement questions (1GP, managed 2GP or unlocked) are multiple choice.
Difficulty mix
Leaderboard
Who's best at Packaging
pass@1 on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide, so read overlapping bars as ties. Open in the full leaderboard →
| # | Model and config | Score pass@1 · 95% CI | Tasks |
|---|---|---|---|
| 1 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 44%95% CI 22–67 | 18 |
| 2 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 22%95% CI 6–44 | 18 |
| 2 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 22%95% CI 6–44 | 18 |
| 2 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 22%95% CI 6–44 | 18 |
| 5 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 19%95% CI 6–35 | 18 |
| 6 | Gemma 4 31B QAT W4A16 · vLLM · effort on | 17%95% CI 0–33 | 18 |
| 6 | Qwen3.6 35B-A3B NVFP4 · vLLM · effort on | 17%95% CI 0–33 | 18 |
| 6 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 17%95% CI 0–33 | 18 |
| 6 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 17%95% CI 0–33 | 18 |
| 10 | Gemma 4 26B-A4B NVFP4 · vLLM · effort on | 11%95% CI 0–28 | 18 |
| 10 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 11%95% CI 0–28 | 18 |
| 10 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 11%95% CI 0–28 | 18 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort highRun so far: 14 of 15 suites · 219 of 272 tasks graded | 41%95% CI 18–65 | 17 |
Tasks
What's in the suite
Solve rate is the mean pass@1 on that task across every finished run, a rough guide to how hard models find it.
| Task | Difficulty | Solve rate |
|---|---|---|
| Configure a new unlocked package in sfdx-project.jsonpackaging-unlocked-project-config | easy | 17% |
| How second-generation managed packaging differs from 1GPpackaging-1gp-vs-2gp | easy | 100% |
| Match packaging IDs to the commands that use thempackaging-package-id-prefixes | easy | 72% |
| Package metadata that depends on unpackaged org metadatapackaging-create-org-dependent | easy | 0% |
| Stop installs of an accidentally promoted managed versionpackaging-lock-promoted-version | easy | 8% |
| Build a promotable package version in CIpackaging-version-create-release-candidate | medium | 8% |
| Build an extension package against the promoted base versionpackaging-dependency-released-build | medium | 67% |
| Create an unlocked package without the project namespacepackaging-create-unlocked-no-namespace | medium | 0% |
| Managed 2GP project with ancestry and test-only metadatapackaging-managed-test-setup | medium | 0% |
| Promote a package version and install it in production from CIpackaging-promote-and-install | medium | 0% |
| Schedule a push upgrade for a wave of subscriber orgspackaging-push-upgrade-schedule | medium | 0% |
| What --skip-validation means for a release pipelinepackaging-skip-validation-consequences | medium | 44% |
| Abandon a released managed 2GP version through ancestrypackaging-abandon-ancestor | hard | 11% |
| Declare multi-level dependencies across unlocked packagespackaging-transitive-dependencies | hard | 0% |
| Move metadata from one unlocked package to another in productionpackaging-migrate-metadata-between-packages | hard | 0% |
| Upload a Managed - Released first-generation package versionpackaging-1gp-managed-released-upload | hard | 8% |
| Which removed managed 2GP components are hard deleted on upgrade?packaging-removed-components-upgrade | hard | 0% |
| Which subscriber upgrades does a 2GP ancestry tree allow?packaging-ancestry-upgrade-paths | hard | 8% |
Work with Leo
Want this for your team?
Leo helps Salesforce teams run AI they own: open-weight models on infrastructure you control, tested on your kind of work before you rely on them. A private benchmark of your shortlist is a fixed fee.
Get a private benchmarkOr email leo@azl.au · azl.au