Suites / docs
Salesforce docs
Navigating the official Salesforce documentation: limits, allocations and platform behaviour an engineer has to look up while designing a solution. Each answer is a short, exact fact and must cite the developer.salesforce.com page that states it; both the fact and the cited page are checked deterministically.
How it's graded
Each answer is a short, exact fact and must cite the developer.salesforce.com page that states it. Both the fact and the cited page are checked deterministically.
Difficulty mix
Leaderboard
Who's best at Salesforce docs
pass@1 on this suite only, with 95% confidence intervals. With 17 tasks the intervals are wide, so read overlapping bars as ties. Open in the full leaderboard →
| # | Model and config | Score pass@1 · 95% CI | Tasks |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 24%95% CI 6–47 | 17 |
| 1 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 24%95% CI 6–47 | 17 |
| 3 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 6%95% CI 0–18 | 17 |
| 3 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 6%95% CI 0–18 | 17 |
| 5 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 4%95% CI 0–12 | 17 |
| 6 | Gemma 4 26B-A4B NVFP4 · vLLM · effort on | 0%95% CI 0–0 | 17 |
| 6 | Gemma 4 31B QAT W4A16 · vLLM · effort on | 0%95% CI 0–0 | 17 |
| 6 | Qwen3.6 35B-A3B NVFP4 · vLLM · effort on | 0%95% CI 0–0 | 17 |
| 6 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 0%95% CI 0–0 | 17 |
| 6 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 0%95% CI 0–0 | 17 |
| 6 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 0%95% CI 0–0 | 17 |
| 6 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 0%95% CI 0–0 | 17 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort highRun so far: 14 of 15 suites · 219 of 272 tasks graded | 29%95% CI 12–53 | 17 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort offRun so far: 2 of 15 suites · 35 of 272 tasks graded | 18%95% CI 0–35 | 17 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort lowRun so far: 2 of 15 suites · 35 of 272 tasks graded | 18%95% CI 0–35 | 17 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort maxRun so far: 2 of 15 suites · 27 of 272 tasks graded | 17%95% CI 0–42 | 12 |
Tasks
What's in the suite
Solve rate is the mean pass@1 on that task across every finished run, a rough guide to how hard models find it.
| Task | Difficulty | Solve rate |
|---|---|---|
| Concurrent long-running API request limitdocs-api-concurrent-long-running | easy | 8% |
| Default Apex callout timeoutdocs-callout-default-timeout | easy | 0% |
| Maximum scratch org durationdocs-scratch-org-max-duration | easy | 31% |
| REST response header with API usagedocs-rest-limit-info-header | easy | 0% |
| Where to enable Lightning Web Securitydocs-lws-enable-setting | easy | 0% |
| After trigger vs after-save flow orderdocs-order-exec-after-trigger-vs-flow | medium | 8% |
| Batch Apex scope maximum with a QueryLocatordocs-batch-querylocator-scope-max | medium | 17% |
| Callout response size limit in async Apexdocs-callout-async-response-size | medium | 8% |
| Default platform event trigger batch sizedocs-pe-trigger-batch-size | medium | 8% |
| Jobs a running Queueable can enqueuedocs-queueable-async-enqueue-limit | medium | 8% |
| Raw CSV size per Bulk API 2.0 ingest jobdocs-bulk2-ingest-upload-size | medium | 0% |
| What LWS uses instead of secure wrappersdocs-lws-distortions | medium | 0% |
| Compute an org's daily API request allocationdocs-api-daily-allocation-calc | hard | 0% |
| Concurrent long-running Apex transactions for a large orgdocs-long-running-apex-concurrency | hard | 0% |
| Cumulative cross-namespace SOQL limitdocs-cross-namespace-soql-limit | hard | 0% |
| Platform event delivery allocation with several subscribersdocs-pe-delivery-allocation-calc | hard | 0% |
| Text length budget in a big object indexdocs-big-object-index-text-length | hard | 0% |
Work with Leo
Want this for your team?
Leo helps Salesforce teams run AI they own: open-weight models on infrastructure you control, tested on your kind of work before you rely on them. A private benchmark of your shortlist is a fixed fee.
Get a private benchmarkOr email leo@azl.au · azl.au