Suites / apex
Apex
General Apex engineering: collections and aggregate SOQL, partial-success DML, triggers and handlers (bulkification, recursion), batch, queueable (chaining, finalizers) and schedulable jobs, platform events, callouts through Named Credentials, JSON, dynamic SOQL, Schema describe, and security at API 67.0 (user mode by default). Each answer is deployed check-only to a clean scratch org at API 67.0 and must pass hidden Apex tests, including 200-record bulk tests. Test-writing tasks are graded by mutation testing: the model's tests must pass on the given implementation and fail on every hidden buggy mutant.
How it's graded
Each answer is deployed as a check-only validation to a clean scratch org together with hidden Apex tests, including 200-record bulk tests and governor-limit assertions. It passes only if it compiles and every test passes. Test-writing tasks are graded by mutation testing: the model’s tests must pass against correct code and fail against seeded bugs.
Difficulty mix
Leaderboard
Who's best at Apex
pass@1 on this suite only, with 95% confidence intervals. With 20 tasks the intervals are wide, so read overlapping bars as ties. Open in the full leaderboard →
| # | Model and config | Score pass@1 · 95% CI | Tasks |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 50%95% CI 30–70 | 20 |
| 1 | Gemma 4 26B-A4B NVFP4 · vLLM · effort on | 50%95% CI 30–70 | 20 |
| 1 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 50%95% CI 30–70 | 20 |
| 4 | Gemma 4 31B QAT W4A16 · vLLM · effort on | 35%95% CI 15–55 | 20 |
| 4 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 35%95% CI 15–55 | 20 |
| 6 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 30%95% CI 10–50 | 20 |
| 7 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 25%95% CI 5–45 | 20 |
| 8 | Qwen3.6 35B-A3B NVFP4 · vLLM · effort on | 15%95% CI 0–30 | 20 |
| 9 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 10%95% CI 0–25 | 20 |
| 9 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 10%95% CI 0–25 | 20 |
| 11 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 7%95% CI 2–13 | 20 |
| 12 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 0%95% CI 0–0 | 20 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort highRun so far: 14 of 15 suites · 219 of 272 tasks graded | 39%95% CI 17–61 | 18 |
Tasks
What's in the suite
Solve rate is the mean pass@1 on that task across every finished run, a rough guide to how hard models find it.
| Task | Difficulty | Solve rate |
|---|---|---|
| Active picklist values via Schema describeapex-picklist-describe | easy | 17% |
| Bulk-safe invocable action for Flowapex-invocable-contact-lookup | easy | 25% |
| Closed-won amount per account with aggregate SOQLapex-won-amount-by-account | easy | 78% |
| Group contacts by email domainapex-contact-email-domain | easy | 53% |
| Nightly schedulable log purge with retention rulesapex-schedulable-log-purge | easy | 25% |
| Payment terms parser with a custom exception hierarchyapex-custom-exception-hierarchy | easy | 0% |
| Bulkify a legacy Contact trigger without changing behaviourapex-bulkify-legacy-trigger | medium | 53% |
| Parse an order payload whose keys are Apex reserved wordsapex-json-reserved-keys | medium | 17% |
| Partial-success lead import with per-row errorsapex-partial-success-lead-import | medium | 42% |
| Publish and subscribe to shipment platform eventsapex-platform-event-shipments | medium | 25% |
| Queueable with a transaction finalizer that logs and retriesapex-queueable-finalizer-retry | medium | 8% |
| REST callout through a Named Credential with a single retryapex-callout-named-credential-retry | medium | 36% |
| Stateful batch that flags overdue invoices and reports totalsapex-batch-stateful-overdue | medium | 25% |
| Write tests that catch bugs in a shipping calculatorapex-mutation-tests-shipping | medium | 58% |
| Chained Queueable that is testable end to endapex-queueable-chain-depth | hard | 0% |
| Injection-safe dynamic SOQL search in user modeapex-dynamic-soql-user-mode | hard | 17% |
| Open-pipeline rollup trigger that survives recursionapex-rollup-trigger-recursion | hard | 17% |
| Upgrade a service to API 67.0 without changing its security behaviourapex-api67-access-modes | hard | 0% |
| Write tests that catch bugs in an Opportunity trigger handlerapex-mutation-tests-won-handler | hard | 0% |
| Write tests with an HttpCalloutMock that pin a REST contractapex-mutation-tests-callout | hard | 33% |
Work with Leo
Want this for your team?
Leo helps Salesforce teams run AI they own: open-weight models on infrastructure you control, tested on your kind of work before you rely on them. A private benchmark of your shortlist is a fixed fee.
Get a private benchmarkOr email leo@azl.au · azl.au