Suites / soql

SOQL

Text-to-SOQL on a realistic B2B sales and service org: filters, relationship queries in both directions, semi- and anti-joins, aggregates, date and fiscal functions, polymorphic relationships, multi-select picklists and null handling. Each answer is executed in a seeded scratch org (orgs/base) and its result set must equal the gold query's (execution accuracy).

20
tasks
Execution
grading
100%
highest c@3 · Claude Haiku 5.5, joint first with 32 others
78
configurations with results

How it's graded

Each query runs against a scratch org seeded with fixed data, and its result set must equal the gold query’s (execution accuracy, as in BIRD and Spider). A query that errors or returns different rows fails.

Difficulty mix

6 easy8 medium6 hard

Leaderboard

Who's best at SOQL

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 20 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at SOQL
Within three attempts (c@3) on SOQL, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
=1Gemini 4 Argon low tie
100%, 95% interval 84–100%
=1Claude Sonnet 5 low tie
100%, 95% interval 84–100%
=1GPT-6 Astra medium tie
100%, 95% interval 84–100%
=1Grok 4.6 high tie
100%, 95% interval 84–100%
=1DeepSeek V4 Pro high tie
100%, 95% interval 84–100%
=1MiMo V2.6 Flash thinking on tie
100%, 95% interval 84–100%
=1Mistral Large 4 high tie
100%, 95% interval 84–100%
=8Nemotron 3 Ultra thinking on tie
95%, 95% interval 76–99%
=8Muse Wonder low tie
95%, 95% interval 76–99%
=8Inkling Small high tie
95%, 95% interval 76–99%
=8Qwen3.8 Flash-Next thinking on tie
95%, 95% interval 76–99%
=12GLM-5.3 low tie
90%, 95% interval 70–97%
=12Laguna S 2.1 thinking on tie
90%, 95% interval 70–97%
14Ling 3.0 Flash thinking on
75%, 95% interval 53–89%
    Show the numbers
    ModelScore95% CI
    Gemini 4 Argon low100%84–100
    Claude Sonnet 5 low100%84–100
    GPT-6 Astra medium100%84–100
    Grok 4.6 high100%84–100
    DeepSeek V4 Pro high100%84–100
    MiMo V2.6 Flash thinking on100%84–100
    Mistral Large 4 high100%84–100
    Nemotron 3 Ultra thinking on95%76–99
    Muse Wonder low95%76–99
    Inkling Small high95%76–99
    Qwen3.8 Flash-Next thinking on95%76–99
    GLM-5.3 low90%70–97
    Laguna S 2.1 thinking on90%70–97
    Ling 3.0 Flash thinking on75%53–89
    Local: the best of each lab at SOQL
    Within three attempts (c@3) on SOQL, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    =1Qwen3.8 Flash-Next medium · NVFP4 tie
    100%, 95% interval 84–100%
    =1MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts tie
    100%, 95% interval 84–100%
    =3GLM-5.3 Flash thinking off · EXL3 4.0bpw experts, FP8 dense tie
    95%, 95% interval 76–99%
    =3DeepSeek V4.1 Flash high · EXL3 2.9bpw tie
    95%, 95% interval 76–99%
    5Gemma 4 26B-A4B thinking on · NVFP4
    90%, 95% interval 70–97%
    =6MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    85%, 95% interval 64–95%
    =6Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt) tie
    85%, 95% interval 64–95%
    8Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)
    80%, 95% interval 58–92%
    =9MiniCPM-V 4.7 35B-A3B thinking on · BF16 tie
    65%, 95% interval 43–82%
    =9Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt) tie
    65%, 95% interval 43–82%
      Show the numbers
      ModelScore95% CI
      Qwen3.8 Flash-Next medium · NVFP4100%84–100
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts100%84–100
      GLM-5.3 Flash thinking off · EXL3 4.0bpw experts, FP8 dense95%76–99
      DeepSeek V4.1 Flash high · EXL3 2.9bpw95%76–99
      Gemma 4 26B-A4B thinking on · NVFP490%70–97
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)85%64–95
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)85%64–95
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)80%58–92
      MiniCPM-V 4.7 35B-A3B thinking on · BF1665%43–82
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)65%43–82
      SOQL: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Claude Haiku 5.5 effort high
      100%95% CI 84–100
      		20
      1Claude Haiku 5.5 effort low
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 effort high
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 effort medium
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 effort low
      100%95% CI 84–100
      		20
      1Claude Opus 4.8 thinking off
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort high
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort medium
      100%95% CI 84–100
      		20
      1Claude Opus 5.5 effort low
      100%95% CI 84–100
      		20
      1Claude Sonnet 5 effort high
      100%95% CI 84–100
      		20
      1Claude Sonnet 5 effort medium
      100%95% CI 84–100
      		20
      1Claude Sonnet 5 thinking off
      100%95% CI 84–100
      		20
      1Claude Sonnet 5 effort low
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort high
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort low
      100%95% CI 84–100
      		20
      1Claude Sonnet 5.5 effort medium
      100%95% CI 84–100
      		20
      1DeepSeek V4 Pro FP8 · effort high
      100%95% CI 84–100
      		20
      1Gemini 3.8 Flash effort medium
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort medium
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort low
      100%95% CI 84–100
      		20
      1Gemini 4 Argon effort high
      100%95% CI 84–100
      		20
      1GPT-5.6 Sol effort medium
      100%95% CI 84–100
      		20
      1GPT-6 Astra effort medium
      100%95% CI 84–100
      		20
      1GPT-6 Astra effort high
      100%95% CI 84–100
      		20
      1GPT-6 Luna effort medium
      100%95% CI 84–100
      		20
      1GPT-6.1 Sol effort high
      100%95% CI 84–100
      		20
      1Grok 4.6 effort high
      100%95% CI 84–100
      		20
      1Grok 4.7 effort high
      100%95% CI 84–100
      		20
      1MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      100%95% CI 84–100
      		20
      1MiMo V2.6 Flash FP8 · thinking on
      100%95% CI 84–100
      		20
      1MiMo V2.6 Pro FP8 · thinking on
      100%95% CI 84–100
      		20
      1Mistral Large 4 effort high
      100%95% CI 84–100
      		20
      1Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      100%95% CI 84–100
      		20
      34Claude Fable 5.1 effort high
      95%95% CI 76–99
      		20
      34Claude Haiku 5.5 effort medium
      95%95% CI 76–99
      		20
      34DeepSeek V4.1 Flash FP8 · effort high
      95%95% CI 76–99
      		20
      34DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      95%95% CI 76–99
      		20
      34DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      95%95% CI 76–99
      		20
      34DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      95%95% CI 76–99
      		20
      34GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      95%95% CI 76–99
      		20
      34GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      95%95% CI 76–99
      		20
      34Inkling Small FP8 · effort high
      95%95% CI 76–99
      		20
      34Muse Wonder effort medium
      95%95% CI 76–99
      		20
      34Muse Wonder effort max
      95%95% CI 76–99
      		20
      34Muse Wonder effort low
      95%95% CI 76–99
      		20
      34Muse Wonder effort high
      95%95% CI 76–99
      		20
      34Nemotron 3 Ultra thinking on
      95%95% CI 76–99
      		20
      34Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      95%95% CI 76–99
      		20
      34Qwen3.8 Flash-Next thinking on
      95%95% CI 76–99
      		20
      50DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      90%95% CI 70–97
      		20
      50Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      90%95% CI 70–97
      		20
      50GLM-5.3 FP8 · effort low
      90%95% CI 70–97
      		20
      50GLM-5.3 Flash FP8 · effort high
      90%95% CI 70–97
      		20
      50GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      90%95% CI 70–97
      		20
      50Laguna S 2.1 FP4 · thinking on
      90%95% CI 70–97
      		20
      50Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      90%95% CI 70–97
      		20
      57GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      85%95% CI 64–95
      		20
      57MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      85%95% CI 64–95
      		20
      57Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      85%95% CI 64–95
      		20
      57Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      85%95% CI 64–95
      		20
      57Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      85%95% CI 64–95
      		20
      62Claude Haiku 4.5 thinking off
      80%95% CI 58–92
      		20
      62Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      80%95% CI 58–92
      		20
      62Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      80%95% CI 58–92
      		20
      62Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      80%95% CI 58–92
      		20
      66Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      78%95% CI 56–90
      		20
      67Gemma 4 31B QAT W4A16 · vLLM · thinking on
      75%95% CI 53–89
      		20
      67Ling 3.0 Flash BF16 · thinking on
      75%95% CI 53–89
      		20
      69Claude Haiku 4.5 thinking on
      70%95% CI 48–85
      		20
      69Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      70%95% CI 48–85
      		20
      69Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      70%95% CI 48–85
      		20
      72MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      65%95% CI 43–82
      		20
      72MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      65%95% CI 43–82
      		20
      72Nemotron 3.5 Lightning NVFP4 · thinking on
      65%95% CI 43–82
      		20
      72Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      65%95% CI 43–82
      		20
      72Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      65%95% CI 43–82
      		20
      72Qwen3.8 27B Splash 4-bit · Splash · effort low
      65%95% CI 43–82
      		20
      72Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      65%95% CI 43–82
      		20

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      SOQL tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      Accounts missing billing country or industrysoql-accounts-missing-country-or-industryeasyeasy
      95%
      Count open German web leads for a dashboard tilesoql-open-german-web-leads-counteasymedium
      66%
      Energy-sector wins in calendar 2025soql-energy-wins-2025easyeasy
      93%
      Financial-services accounts in Germany and the UKsoql-banking-insurance-de-ukeasyeasy
      100%
      In-transit shipments with their account namesoql-in-transit-shipmentseasyeasy
      92%
      Managers at healthcare accountssoql-healthcare-manager-contactseasyeasy
      97%
      2025 closed-won business by industrysoql-won-2025-by-industrymediumeasy
      68%
      Accounts with open high-priority casessoql-accounts-open-high-casesmediumeasy
      99%
      Customers with no wins in 2025soql-customers-without-2025-winsmediumeasy
      92%
      Five largest customers by annual revenuesoql-top-customers-by-revenuemediumeasy
      99%
      German customers with their open shipments nestedsoql-german-customers-open-shipmentsmediummedium
      39%
      Open tasks related to opportunitiessoql-open-tasks-on-opportunitiesmediumeasy
      94%
      Products whose code starts with SVC_soql-service-product-codesmediumeasy
      72%
      Responders to an event campaignsoql-summit-respondersmediumeasy
      72%
      Activity log with type-specific related-record fieldssoql-march-2025-activity-typeofhardeasy
      68%
      Bookings per fiscal quarter with a non-calendar fiscal yearsoql-fy2026-bookings-by-quarterhardmedium
      65%
      Contacts anywhere in an account hierarchysoql-globex-group-contactshardeasy
      72%
      Hardware deals without services attachedsoql-hardware-without-services-oppshardeasy
      99%
      Open shipments needing special handling (multi-select picklist)soql-open-shipments-handlinghardeasy
      70%
      Parent accounts of German subsidiariessoql-parents-of-german-accountshardhard
      19%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au