Suites / permissions

Permissions & access

Least-privilege access design: permission sets (object, field, tab, app, Apex class, user and custom permissions), session-based permission sets, permission set groups with muting, fixing over-permissive access, and Apex that checks access (custom permissions, stripInaccessible, UserRecordAccess) under API 67.0's user-mode default. Metadata answers are deployed check-only to a scratch org with hidden custom objects; hidden Apex tests create Minimum Access users, assign the answer's permission sets or groups and assert effective access with System.runAs. A few knowledge tasks cover how profiles, permission sets, groups, muting and sharing combine.

15
tasks
Execution + validators
grading
100%
highest c@3 · Claude Opus 5.5, joint first with 5 others
78
configurations with results

How it's graded

Permission sets and groups are deployed as a check-only validation to a scratch org. Hidden Apex tests create Minimum Access users, assign the answer’s permission sets or groups and assert effective access with System.runAs. A few knowledge tasks are multiple choice or have a short exact answer.

Difficulty mix

4 easy6 medium5 hard

Leaderboard

Who's best at Permissions & access

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 15 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at Permissions & access
Within three attempts (c@3) on Permissions & access, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
=1GPT-6 Astra medium tie
100%, 95% interval 80–100%
=1Claude Opus 5.5 low tie
100%, 95% interval 80–100%
=1Gemini 4 Argon high tie
100%, 95% interval 80–100%
=4Muse Wonder low tie
93%, 95% interval 70–99%
=4Grok 4.6 high tie
93%, 95% interval 70–99%
6DeepSeek V4.1 Flash high
87%, 95% interval 62–96%
=7GLM-5.3 Flash high tie
80%, 95% interval 55–93%
=7MiMo V2.6 Pro thinking on tie
80%, 95% interval 55–93%
=9Nemotron 3 Ultra thinking on tie
67%, 95% interval 42–85%
=9Inkling Small high tie
67%, 95% interval 42–85%
=9Mistral Large 4 high tie
67%, 95% interval 42–85%
12Qwen3.8 Flash-Next thinking on
60%, 95% interval 36–80%
13Laguna S 2.1 thinking on
47%, 95% interval 25–70%
14Ling 3.0 Flash thinking on
33%, 95% interval 15–58%
    Show the numbers
    ModelScore95% CI
    GPT-6 Astra medium100%80–100
    Claude Opus 5.5 low100%80–100
    Gemini 4 Argon high100%80–100
    Muse Wonder low93%70–99
    Grok 4.6 high93%70–99
    DeepSeek V4.1 Flash high87%62–96
    GLM-5.3 Flash high80%55–93
    MiMo V2.6 Pro thinking on80%55–93
    Nemotron 3 Ultra thinking on67%42–85
    Inkling Small high67%42–85
    Mistral Large 4 high67%42–85
    Qwen3.8 Flash-Next thinking on60%36–80
    Laguna S 2.1 thinking on47%25–70
    Ling 3.0 Flash thinking on33%15–58
    Local: the best of each lab at Permissions & access
    Within three attempts (c@3) on Permissions & access, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    =1GLM-5.3 Flash low · EXL3 4.0bpw experts, FP8 dense tie
    87%, 95% interval 62–96%
    =1DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts tie
    87%, 95% interval 62–96%
    3MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts
    73%, 95% interval 48–89%
    =4Qwen3.8 27B low · AWQ-INT4 tie
    67%, 95% interval 42–85%
    =4MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    67%, 95% interval 42–85%
    6Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)
    53%, 95% interval 30–75%
    =7Gemma 4 31B thinking on · QAT W4A16 tie
    47%, 95% interval 25–70%
    =7Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt) tie
    47%, 95% interval 25–70%
    9Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)
    40%, 95% interval 20–64%
    10MiniCPM-V 4.7 35B-A3B thinking on · BF16
    33%, 95% interval 15–58%
      Show the numbers
      ModelScore95% CI
      GLM-5.3 Flash low · EXL3 4.0bpw experts, FP8 dense87%62–96
      DeepSeek V4.1 Flash 1017 high · Native FP8, FP4 experts87%62–96
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts73%48–89
      Qwen3.8 27B low · AWQ-INT467%42–85
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)67%42–85
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)53%30–75
      Gemma 4 31B thinking on · QAT W4A1647%25–70
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)47%25–70
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)40%20–64
      MiniCPM-V 4.7 35B-A3B thinking on · BF1633%15–58
      Permissions & access: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Claude Opus 5.5 effort high
      100%95% CI 80–100
      		15
      1Claude Opus 5.5 effort medium
      100%95% CI 80–100
      		15
      1Claude Opus 5.5 effort low
      100%95% CI 80–100
      		15
      1Gemini 4 Argon effort high
      100%95% CI 80–100
      		15
      1GPT-6 Astra effort medium
      100%95% CI 80–100
      		15
      1GPT-6.1 Sol effort high
      100%95% CI 80–100
      		15
      7Claude Fable 5.1 effort high
      93%95% CI 70–99
      		15
      7Claude Opus 4.8 effort high
      93%95% CI 70–99
      		15
      7Claude Opus 4.8 effort medium
      93%95% CI 70–99
      		15
      7Claude Sonnet 5.5 effort high
      93%95% CI 70–99
      		15
      7Claude Sonnet 5.5 effort low
      93%95% CI 70–99
      		15
      7Claude Sonnet 5.5 effort medium
      93%95% CI 70–99
      		15
      7Gemini 4 Argon effort medium
      93%95% CI 70–99
      		15
      7Gemini 4 Argon effort low
      93%95% CI 70–99
      		15
      7GPT-5.6 Sol effort medium
      93%95% CI 70–99
      		15
      7GPT-6 Astra effort high
      93%95% CI 70–99
      		15
      7Grok 4.6 effort high
      93%95% CI 70–99
      		15
      7Grok 4.7 effort high
      93%95% CI 70–99
      		15
      7Muse Wonder effort max
      93%95% CI 70–99
      		15
      7Muse Wonder effort low
      93%95% CI 70–99
      		15
      21Claude Haiku 5.5 effort medium
      87%95% CI 62–96
      		15
      21Claude Haiku 5.5 effort high
      87%95% CI 62–96
      		15
      21Claude Haiku 5.5 effort low
      87%95% CI 62–96
      		15
      21Claude Opus 4.8 effort low
      87%95% CI 62–96
      		15
      21Claude Opus 4.8 thinking off
      87%95% CI 62–96
      		15
      21Claude Sonnet 5 thinking off
      87%95% CI 62–96
      		15
      21DeepSeek V4 Pro FP8 · effort high
      87%95% CI 62–96
      		15
      21DeepSeek V4.1 Flash FP8 · effort high
      87%95% CI 62–96
      		15
      21DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      87%95% CI 62–96
      		15
      21Gemini 3.8 Flash effort medium
      87%95% CI 62–96
      		15
      21GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      87%95% CI 62–96
      		15
      21GPT-6 Luna effort medium
      87%95% CI 62–96
      		15
      21Muse Wonder effort medium
      87%95% CI 62–96
      		15
      21Muse Wonder effort high
      87%95% CI 62–96
      		15
      35Claude Sonnet 5 effort high
      80%95% CI 55–93
      		15
      35Claude Sonnet 5 effort medium
      80%95% CI 55–93
      		15
      35Claude Sonnet 5 effort low
      80%95% CI 55–93
      		15
      35DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      80%95% CI 55–93
      		15
      35DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      80%95% CI 55–93
      		15
      35GLM-5.3 Flash FP8 · effort high
      80%95% CI 55–93
      		15
      35MiMo V2.6 Pro FP8 · thinking on
      80%95% CI 55–93
      		15
      42Claude Haiku 4.5 thinking on
      73%95% CI 48–89
      		15
      42GLM-5.3 FP8 · effort low
      73%95% CI 48–89
      		15
      42GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      73%95% CI 48–89
      		15
      42MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      73%95% CI 48–89
      		15
      42MiMo V2.6 Flash FP8 · thinking on
      73%95% CI 48–89
      		15
      47GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      67%95% CI 42–85
      		15
      47GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      67%95% CI 42–85
      		15
      47Inkling Small FP8 · effort high
      67%95% CI 42–85
      		15
      47MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      67%95% CI 42–85
      		15
      47Mistral Large 4 effort high
      67%95% CI 42–85
      		15
      47Nemotron 3 Ultra thinking on
      67%95% CI 42–85
      		15
      47Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      67%95% CI 42–85
      		15
      47Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      67%95% CI 42–85
      		15
      55DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      60%95% CI 36–80
      		15
      55Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      60%95% CI 36–80
      		15
      55Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      60%95% CI 36–80
      		15
      55Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      60%95% CI 36–80
      		15
      55Qwen3.8 Flash-Next thinking on
      60%95% CI 36–80
      		15
      60Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      53%95% CI 30–75
      		15
      60Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      53%95% CI 30–75
      		15
      62Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      50%95% CI 27–73
      		15
      63Claude Haiku 4.5 thinking off
      47%95% CI 25–70
      		15
      63Gemma 4 31B QAT W4A16 · vLLM · thinking on
      47%95% CI 25–70
      		15
      63Laguna S 2.1 FP4 · thinking on
      47%95% CI 25–70
      		15
      63Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      47%95% CI 25–70
      		15
      63Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      47%95% CI 25–70
      		15
      63Qwen3.8 27B Splash 4-bit · Splash · effort low
      47%95% CI 25–70
      		15
      69Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      40%95% CI 20–64
      		15
      69Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      40%95% CI 20–64
      		15
      69Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      40%95% CI 20–64
      		15
      69Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      40%95% CI 20–64
      		15
      69Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      40%95% CI 20–64
      		15
      74Ling 3.0 Flash BF16 · thinking on
      33%95% CI 15–58
      		15
      74MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      33%95% CI 15–58
      		15
      76MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      20%95% CI 7–45
      		15
      76Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      20%95% CI 7–45
      		15
      78Nemotron 3.5 Lightning NVFP4 · thinking on
      7%95% CI 1–30
      		15

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      Permissions & access tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      App and tab visibility in a permission setpermissions-tab-app-visibilityeasyhard
      33%
      Integration user permission set for Apex RESTpermissions-integration-api-accesseasymedium
      51%
      Read-only claims viewer permission setpermissions-readonly-claims-viewereasymedium
      63%
      Session-based permission set for refund elevationpermissions-session-based-elevationeasymedium
      53%
      Auditors see every review but edit only their ownpermissions-view-all-not-modify-allmediumeasy
      68%
      Fix an over-permissive technician permission setpermissions-fix-over-permissivemediumeasy
      94%
      Gate large discounts with a custom permissionpermissions-custom-permission-apexmediummedium
      62%
      Invoice clerk access with a required fieldpermissions-required-field-flsmediumhard
      25%
      Object permissions versus record sharingpermissions-sharing-vs-object-permsmediumeasy
      85%
      What must stay on the profilepermissions-profile-only-settingsmediummedium
      50%
      Bulk-safe "which records can I edit" checkpermissions-bulk-editable-idshardmedium
      39%
      Directory query that drops unreadable fields at API 67.0permissions-strip-inaccessible-v67hardmedium
      41%
      Effective access with a muted group plus direct assignmentspermissions-effective-access-mutinghardeasy
      92%
      Permission set group with a muting permission setpermissions-psg-mutinghardhard
      15%
      What sharing keywords enforce at API 66.0 and 67.0permissions-with-sharing-api-versionshardhard
      8%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au