Suites / api
Salesforce APIs
Integration work against Salesforce's APIs: REST sObject, query, composite, composite graph and sObject Collections resources, Bulk API 2.0 ingest and query jobs, Tooling API, User Interface API, platform events, custom Apex REST endpoints and OAuth 2.0 token requests. Answers are raw HTTP requests, checked deterministically for method, path, query, headers and body (JSON, form-encoded or CSV), plus a few questions on API behaviour and choice.
How it's graded
Answers are raw HTTP requests, checked deterministically for method, path, query, headers and body (JSON, form-encoded or CSV). A few questions cover API behaviour and choosing the right API.
Difficulty mix
Leaderboard
Who's best at Salesforce APIs
pass@1 on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide, so read overlapping bars as ties. Open in the full leaderboard →
| # | Model and config | Score pass@1 · 95% CI | Tasks |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 78%95% CI 56–94 | 18 |
| 2 | Gemma 4 31B QAT W4A16 · vLLM · effort on | 50%95% CI 28–72 | 18 |
| 3 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 44%95% CI 22–67 | 18 |
| 4 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 41%95% CI 20–61 | 18 |
| 5 | Gemma 4 26B-A4B NVFP4 · vLLM · effort on | 39%95% CI 17–61 | 18 |
| 5 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 39%95% CI 17–61 | 18 |
| 5 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 39%95% CI 17–61 | 18 |
| 5 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 39%95% CI 17–61 | 18 |
| 5 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 39%95% CI 17–61 | 18 |
| 10 | Qwen3.6 35B-A3B NVFP4 · vLLM · effort on | 33%95% CI 11–56 | 18 |
| 10 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 33%95% CI 11–56 | 18 |
| 12 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 28%95% CI 11–50 | 18 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort maxRun so far: 2 of 15 suites · 27 of 272 tasks graded | 73%95% CI 47–93 | 15 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort offRun so far: 2 of 15 suites · 35 of 272 tasks graded | 56%95% CI 33–78 | 18 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort highRun so far: 14 of 15 suites · 219 of 272 tasks graded | 39%95% CI 17–61 | 18 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort lowRun so far: 2 of 15 suites · 35 of 272 tasks graded | 39%95% CI 17–61 | 18 |
Tasks
What's in the suite
Solve rate is the mean pass@1 on that task across every finished run, a rough guide to how hard models find it.
| Task | Difficulty | Solve rate |
|---|---|---|
| Close an opportunity and delete a duplicate contactapi-sobject-update-delete | easy | 100% |
| Create a Case with the REST APIapi-sobject-create-case | easy | 100% |
| Pick the API for a nightly 5-million-row loadapi-choose-bulk-for-large-load | easy | 100% |
| Pre-sync checks with describe and limitsapi-describe-and-limits | easy | 92% |
| Publish a platform event from an external systemapi-platform-event-publish | easy | 83% |
| Query Apex code coverage with the Tooling APIapi-tooling-coverage-query | easy | 8% |
| Call a managed-package Apex REST endpointapi-apexrest-namespaced-post | medium | 33% |
| Create and update records with sObject Collectionsapi-collections-create-update | medium | 19% |
| Fetch a record with UI API fields and optionalFieldsapi-uiapi-record-optional-fields | medium | 0% |
| Page through a delta query that includes deleted recordsapi-query-paging-queryall | medium | 0% |
| Request a token with the OAuth 2.0 client credentials flowapi-oauth-client-credentials | medium | 97% |
| Run anonymous Apex through the Tooling APIapi-tooling-execute-anonymous | medium | 0% |
| Upsert an account by external ID with its parentapi-upsert-external-id | medium | 44% |
| Bulk API 2.0 export including deleted recordsapi-bulk2-query-export | hard | 17% |
| Bulk API 2.0 upsert with a parent external IDapi-bulk2-ingest-upsert | hard | 17% |
| Composite request with an implicit dependencyapi-composite-implicit-dependency | hard | 0% |
| Independent all-or-nothing orders in one callapi-composite-graph-orders | hard | 8% |
| Interpret external-ID upsert responsesapi-upsert-response-semantics | hard | 33% |
Work with Leo
Want this for your team?
Leo helps Salesforce teams run AI they own: open-weight models on infrastructure you control, tested on your kind of work before you rely on them. A private benchmark of your shortlist is a fixed fee.
Get a private benchmarkOr email leo@azl.au · azl.au