Suites / permissions
Permissions & access
Least-privilege access design: permission sets (object, field, tab, app, Apex class, user and custom permissions), session-based permission sets, permission set groups with muting, fixing over-permissive access, and Apex that checks access (custom permissions, stripInaccessible, UserRecordAccess) under API 67.0's user-mode default. Metadata answers are deployed check-only to a scratch org with hidden custom objects; hidden Apex tests create Minimum Access users, assign the answer's permission sets or groups and assert effective access with System.runAs. A few knowledge tasks cover how profiles, permission sets, groups, muting and sharing combine.
How it's graded
Permission sets and groups are deployed as a check-only validation to a scratch org. Hidden Apex tests create Minimum Access users, assign the answer’s permission sets or groups and assert effective access with System.runAs. A few knowledge tasks are multiple choice or have a short exact answer.
Difficulty mix
Leaderboard
Who's best at Permissions & access
pass@1 on this suite only, with 95% confidence intervals. With 15 tasks the intervals are wide, so read overlapping bars as ties. Open in the full leaderboard →
| # | Model and config | Score pass@1 · 95% CI | Tasks |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 67%95% CI 40–87 | 15 |
| 2 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 47%95% CI 20–73 | 15 |
| 3 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 40%95% CI 13–67 | 15 |
| 3 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 40%95% CI 13–67 | 15 |
| 5 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 33%95% CI 13–60 | 15 |
| 6 | Gemma 4 26B-A4B NVFP4 · vLLM · effort on | 27%95% CI 7–53 | 15 |
| 6 | Gemma 4 31B QAT W4A16 · vLLM · effort on | 27%95% CI 7–53 | 15 |
| 6 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 27%95% CI 7–49 | 15 |
| 6 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 27%95% CI 7–53 | 15 |
| 10 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 20%95% CI 0–40 | 15 |
| 11 | Qwen3.6 35B-A3B NVFP4 · vLLM · effort on | 13%95% CI 0–33 | 15 |
| 11 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 13%95% CI 0–33 | 15 |
| – | GLM-5.3 Flash partialEXL3 4.0bpw · vLLM + ExLlamaV3 · effort highRun so far: 14 of 15 suites · 219 of 272 tasks graded | 40%95% CI 20–67 | 15 |
Tasks
What's in the suite
Solve rate is the mean pass@1 on that task across every finished run, a rough guide to how hard models find it.
| Task | Difficulty | Solve rate |
|---|---|---|
| App and tab visibility in a permission setpermissions-tab-app-visibility | easy | 0% |
| Integration user permission set for Apex RESTpermissions-integration-api-access | easy | 19% |
| Read-only claims viewer permission setpermissions-readonly-claims-viewer | easy | 33% |
| Session-based permission set for refund elevationpermissions-session-based-elevation | easy | 25% |
| Auditors see every review but edit only their ownpermissions-view-all-not-modify-all | medium | 42% |
| Fix an over-permissive technician permission setpermissions-fix-over-permissive | medium | 100% |
| Gate large discounts with a custom permissionpermissions-custom-permission-apex | medium | 25% |
| Invoice clerk access with a required fieldpermissions-required-field-fls | medium | 0% |
| Object permissions versus record sharingpermissions-sharing-vs-object-perms | medium | 75% |
| What must stay on the profilepermissions-profile-only-settings | medium | 31% |
| Bulk-safe "which records can I edit" checkpermissions-bulk-editable-ids | hard | 8% |
| Directory query that drops unreadable fields at API 67.0permissions-strip-inaccessible-v67 | hard | 25% |
| Effective access with a muted group plus direct assignmentspermissions-effective-access-muting | hard | 92% |
| Permission set group with a muting permission setpermissions-psg-muting | hard | 0% |
| What sharing keywords enforce at API 66.0 and 67.0permissions-with-sharing-api-versions | hard | 0% |
Work with Leo
Want this for your team?
Leo helps Salesforce teams run AI they own: open-weight models on infrastructure you control, tested on your kind of work before you rely on them. A private benchmark of your shortlist is a fixed fee.
Get a private benchmarkOr email leo@azl.au · azl.au