Suites / apex

Apex

General Apex engineering: collections and aggregate SOQL, partial-success DML, triggers and handlers (bulkification, recursion), batch, queueable (chaining, finalizers) and schedulable jobs, platform events, callouts through Named Credentials, JSON, dynamic SOQL, Schema describe, and security at API 67.0 (user mode by default). Each answer is deployed check-only to a clean scratch org at API 67.0 and must pass hidden Apex tests, including 200-record bulk tests. Test-writing tasks are graded by mutation testing: the model's tests must pass on the given implementation and fail on every hidden buggy mutant.

20
tasks
Execution
grading
100%
highest c@3 · Claude Fable 5.1, joint first with 16 others
78
configurations with results

How it's graded

Each answer is deployed as a check-only validation to a clean scratch org together with hidden Apex tests, including 200-record bulk tests and governor-limit assertions. It passes only if it compiles and every test passes. Test-writing tasks are graded by mutation testing: the model’s tests must pass against correct code and fail against seeded bugs.

Difficulty mix

6 easy8 medium6 hard

Leaderboard

Who's best at Apex

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 20 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at Apex
Within three attempts (c@3) on Apex, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
=1Gemini 4 Argon low tie
100%, 95% interval 84–100%
=1GPT-6 Astra medium tie
100%, 95% interval 84–100%
=1Claude Sonnet 5.5 medium tie
100%, 95% interval 84–100%
=1Muse Wonder medium tie
100%, 95% interval 84–100%
=1Grok 4.6 high tie
100%, 95% interval 84–100%
6DeepSeek V4.1 Flash high
95%, 95% interval 76–99%
=7GLM-5.3 Flash high tie
90%, 95% interval 70–97%
=7MiMo V2.6 Flash thinking on tie
90%, 95% interval 70–97%
9Inkling Small high
80%, 95% interval 58–92%
=10Nemotron 3 Ultra thinking on tie
75%, 95% interval 53–89%
=10Ling 3.0 Flash thinking on tie
75%, 95% interval 53–89%
=10Mistral Large 4 high tie
75%, 95% interval 53–89%
13Laguna S 2.1 thinking on
55%, 95% interval 34–74%
14Qwen3.8 Flash-Next thinking on
30%, 95% interval 15–52%
    Show the numbers
    ModelScore95% CI
    Gemini 4 Argon low100%84–100
    GPT-6 Astra medium100%84–100
    Claude Sonnet 5.5 medium100%84–100
    Muse Wonder medium100%84–100
    Grok 4.6 high100%84–100
    DeepSeek V4.1 Flash high95%76–99
    GLM-5.3 Flash high90%70–97
    MiMo V2.6 Flash thinking on90%70–97
    Inkling Small high80%58–92
    Nemotron 3 Ultra thinking on75%53–89
    Ling 3.0 Flash thinking on75%53–89
    Mistral Large 4 high75%53–89
    Laguna S 2.1 thinking on55%34–74
    Qwen3.8 Flash-Next thinking on30%15–52
    Local: the best of each lab at Apex
    Within three attempts (c@3) on Apex, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    1DeepSeek V4.1 Flash high · EXL3 2.9bpw
    95%, 95% interval 76–99%
    2GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense
    90%, 95% interval 70–97%
    3MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts
    85%, 95% interval 64–95%
    4Qwen3.8 Flash-Next medium · NVFP4
    80%, 95% interval 58–92%
    5Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)
    75%, 95% interval 53–89%
    6Gemma 4 31B thinking on · QAT W4A16
    70%, 95% interval 48–85%
    7Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)
    60%, 95% interval 39–78%
    8MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)
    55%, 95% interval 34–74%
    9MiniCPM-V 4.7 35B-A3B thinking on · BF16
    50%, 95% interval 30–70%
    10Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)
    40%, 95% interval 22–61%
      Show the numbers
      ModelScore95% CI
      DeepSeek V4.1 Flash high · EXL3 2.9bpw95%76–99
      GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense90%70–97
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts85%64–95
      Qwen3.8 Flash-Next medium · NVFP480%58–92
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)75%53–89
      Gemma 4 31B thinking on · QAT W4A1670%48–85
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)60%39–78
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)55%34–74
      MiniCPM-V 4.7 35B-A3B thinking on · BF1650%30–70
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)40%22–61
      Apex: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Claude Fable 5.1 effort high
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 effort medium
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort high
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort medium
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort low
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort high
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort medium
      100%95% CI 84–100
      		20
      1Gemini 3.8 Flash effort medium
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort low
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort high
      100%95% CI 84–100
      		20
      1GPT-6 Astra effort medium
      100%95% CI 84–100
      		20
      1GPT-6 Astra effort high
      100%95% CI 84–100
      		20
      1GPT-6.1 Sol effort high
      100%95% CI 84–100
      		20
      1Grok 4.6 effort high
      100%95% CI 84–100
      		20
      1Grok 4.7 effort high
      100%95% CI 84–100
      		20
      1Muse Wonder effort medium
      100%95% CI 84–100
      		20
      1Muse Wonder effort max
      100%95% CI 84–100
      		20
      18Claude Opus 4.8 effort high
      95%95% CI 76–99
      		20
      18Claude Sonnet 5.5 effort low
      95%95% CI 76–99
      		20
      18DeepSeek V4.1 Flash FP8 · effort high
      95%95% CI 76–99
      		20
      18DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      95%95% CI 76–99
      		20
      18Gemini 4 Argon effort medium
      95%95% CI 76–99
      		20
      18GPT-5.6 Sol effort medium
      95%95% CI 76–99
      		20
      18Muse Wonder effort low
      95%95% CI 76–99
      		20
      25Claude Haiku 5.5 effort medium
      90%95% CI 70–97
      		20
      25Claude Haiku 5.5 effort high
      90%95% CI 70–97
      		20
      25Claude Haiku 5.5 effort low
      90%95% CI 70–97
      		20
      25Claude Opus 4.8 effort low
      90%95% CI 70–97
      		20
      25Claude Opus 4.8 thinking off
      90%95% CI 70–97
      		20
      25Claude Sonnet 5 effort high
      90%95% CI 70–97
      		20
      25Claude Sonnet 5 effort medium
      90%95% CI 70–97
      		20
      25Claude Sonnet 5 effort low
      90%95% CI 70–97
      		20
      25DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      90%95% CI 70–97
      		20
      25DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      90%95% CI 70–97
      		20
      25GLM-5.3 Flash FP8 · effort high
      90%95% CI 70–97
      		20
      25GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      90%95% CI 70–97
      		20
      25MiMo V2.6 Flash FP8 · thinking on
      90%95% CI 70–97
      		20
      25MiMo V2.6 Pro FP8 · thinking on
      90%95% CI 70–97
      		20
      25Muse Wonder effort high
      90%95% CI 70–97
      		20
      40Claude Sonnet 5 thinking off
      85%95% CI 64–95
      		20
      40GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      85%95% CI 64–95
      		20
      40GPT-6 Luna effort medium
      85%95% CI 64–95
      		20
      40MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      85%95% CI 64–95
      		20
      44DeepSeek V4 Pro FP8 · effort high
      80%95% CI 58–92
      		20
      44GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      80%95% CI 58–92
      		20
      44Inkling Small FP8 · effort high
      80%95% CI 58–92
      		20
      44Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      80%95% CI 58–92
      		20
      48Claude Haiku 4.5 thinking on
      75%95% CI 53–89
      		20
      48Claude Haiku 4.5 thinking off
      75%95% CI 53–89
      		20
      48GLM-5.3 FP8 · effort low
      75%95% CI 53–89
      		20
      48Ling 3.0 Flash BF16 · thinking on
      75%95% CI 53–89
      		20
      48Mistral Large 4 effort high
      75%95% CI 53–89
      		20
      48Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      75%95% CI 53–89
      		20
      48Nemotron 3 Ultra thinking on
      75%95% CI 53–89
      		20
      55Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      70%95% CI 48–85
      		20
      55Gemma 4 31B QAT W4A16 · vLLM · thinking on
      70%95% CI 48–85
      		20
      55Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      70%95% CI 48–85
      		20
      55Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      70%95% CI 48–85
      		20
      59DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      60%95% CI 39–78
      		20
      59Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      60%95% CI 39–78
      		20
      59Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      60%95% CI 39–78
      		20
      59Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      60%95% CI 39–78
      		20
      59Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      60%95% CI 39–78
      		20
      59Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      60%95% CI 39–78
      		20
      65Laguna S 2.1 FP4 · thinking on
      55%95% CI 34–74
      		20
      65MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      55%95% CI 34–74
      		20
      65Qwen3.8 27B Splash 4-bit · Splash · effort low
      55%95% CI 34–74
      		20
      68GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      50%95% CI 30–70
      		20
      68MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      50%95% CI 30–70
      		20
      70Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      48%95% CI 28–68
      		20
      71Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      45%95% CI 26–66
      		20
      71Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      45%95% CI 26–66
      		20
      73Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      40%95% CI 22–61
      		20
      73Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      40%95% CI 22–61
      		20
      75MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      30%95% CI 15–52
      		20
      75Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      30%95% CI 15–52
      		20
      75Qwen3.8 Flash-Next thinking on
      30%95% CI 15–52
      		20
      78Nemotron 3.5 Lightning NVFP4 · thinking on
      15%95% CI 5–36
      		20

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      Apex tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      Active picklist values via Schema describeapex-picklist-describeeasymedium
      61%
      Bulk-safe invocable action for Flowapex-invocable-contact-lookupeasymedium
      56%
      Closed-won amount per account with aggregate SOQLapex-won-amount-by-accounteasyeasy
      74%
      Group contacts by email domainapex-contact-email-domaineasyeasy
      73%
      Nightly schedulable log purge with retention rulesapex-schedulable-log-purgeeasymedium
      51%
      Payment terms parser with a custom exception hierarchyapex-custom-exception-hierarchyeasymedium
      38%
      Bulkify a legacy Contact trigger without changing behaviourapex-bulkify-legacy-triggermediumeasy
      78%
      Parse an order payload whose keys are Apex reserved wordsapex-json-reserved-keysmediummedium
      42%
      Partial-success lead import with per-row errorsapex-partial-success-lead-importmediumeasy
      73%
      Publish and subscribe to shipment platform eventsapex-platform-event-shipmentsmediumeasy
      67%
      Queueable with a transaction finalizer that logs and retriesapex-queueable-finalizer-retrymediummedium
      57%
      REST callout through a Named Credential with a single retryapex-callout-named-credential-retrymediummedium
      49%
      Stateful batch that flags overdue invoices and reports totalsapex-batch-stateful-overduemediummedium
      48%
      Write tests that catch bugs in a shipping calculatorapex-mutation-tests-shippingmediummedium
      57%
      Chained Queueable that is testable end to endapex-queueable-chain-depthhardhard
      15%
      Injection-safe dynamic SOQL search in user modeapex-dynamic-soql-user-modehardmedium
      49%
      Open-pipeline rollup trigger that survives recursionapex-rollup-trigger-recursionhardmedium
      57%
      Upgrade a service to API 67.0 without changing its security behaviourapex-api67-access-modeshardhard
      23%
      Write tests that catch bugs in an Opportunity trigger handlerapex-mutation-tests-won-handlerhardhard
      9%
      Write tests with an HttpCalloutMock that pin a REST contractapex-mutation-tests-callouthardmedium
      46%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au