Suites / npsp

Nonprofit Success Pack

Engineering on the Nonprofit Success Pack (NPSP) managed packages (npsp, npe01, npo02, npe03, npe4, npe5): the Household Account model, gifts and payments, GAU allocations, soft credits, Enhanced Recurring Donations, affiliations and relationships, TDTM trigger handlers and the BDI data import API. Code answers are deployed to a scratch org with NPSP installed and must pass hidden Apex tests that exercise NPSP's own automation; SOQL answers run against seeded NPSP data; knowledge questions cover NPSP architecture and configuration.

18
tasks
Execution + validators
grading
100%
highest c@3 · Gemini 3.8 Flash, joint first with 3 others
78
configurations with results

How it's graded

Code answers are deployed as a check-only validation to a scratch org with NPSP installed and must pass hidden Apex tests that exercise NPSP’s own automation. SOQL answers run against seeded NPSP data; knowledge questions cover NPSP architecture and configuration.

Difficulty mix

5 easy8 medium5 hard

Leaderboard

Who's best at Nonprofit Success Pack

Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →

API: the best of each lab at Nonprofit Success Pack
Within three attempts (c@3) on Nonprofit Success Pack, on the API board (models reached through a hosted service). Bars whose intervals overlap are not clearly different.
  • 95% interval
=1Gemini 4 Argon medium tie
100%, 95% interval 82–100%
=1Muse Wonder low tie
100%, 95% interval 82–100%
=1Grok 4.6 high tie
100%, 95% interval 82–100%
=4GPT-6 Astra medium tie
94%, 95% interval 74–99%
=4Claude Sonnet 5.5 low tie
94%, 95% interval 74–99%
6DeepSeek V4.1 Flash high
89%, 95% interval 67–97%
7MiMo V2.6 Pro thinking on
78%, 95% interval 55–91%
=8GLM-5.3 Flash high tie
67%, 95% interval 44–84%
=8Mistral Large 4 high tie
67%, 95% interval 44–84%
=10Nemotron 3 Ultra thinking on tie
56%, 95% interval 34–75%
=10Inkling Small high tie
56%, 95% interval 34–75%
=12Ling 3.0 Flash thinking on tie
44%, 95% interval 25–66%
=12Qwen3.8 Flash-Next thinking on tie
44%, 95% interval 25–66%
14Laguna S 2.1 thinking on
39%, 95% interval 20–61%
    Show the numbers
    ModelScore95% CI
    Gemini 4 Argon medium100%82–100
    Muse Wonder low100%82–100
    Grok 4.6 high100%82–100
    GPT-6 Astra medium94%74–99
    Claude Sonnet 5.5 low94%74–99
    DeepSeek V4.1 Flash high89%67–97
    MiMo V2.6 Pro thinking on78%55–91
    GLM-5.3 Flash high67%44–84
    Mistral Large 4 high67%44–84
    Nemotron 3 Ultra thinking on56%34–75
    Inkling Small high56%34–75
    Ling 3.0 Flash thinking on44%25–66
    Qwen3.8 Flash-Next thinking on44%25–66
    Laguna S 2.1 thinking on39%20–61
    Local: the best of each lab at Nonprofit Success Pack
    Within three attempts (c@3) on Nonprofit Success Pack, on the Local board (models served on Forcebench’s own hardware). Bars whose intervals overlap are not clearly different.
    • 95% interval
    1DeepSeek V4.1 Flash high · Native FP8, FP4 experts
    89%, 95% interval 67–97%
    2GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense
    83%, 95% interval 61–94%
    =3Gemma 4 31B thinking on · QAT W4A16 tie
    61%, 95% interval 39–80%
    =3Qwen3.8 Flash-Next medium · NVFP4 tie
    61%, 95% interval 39–80%
    =3Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt) tie
    61%, 95% interval 39–80%
    =6Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt) tie
    56%, 95% interval 34–75%
    =6MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts tie
    56%, 95% interval 34–75%
    =8MiniCPM-V 4.7 35B-A3B thinking on · BF16 tie
    50%, 95% interval 29–71%
    =8MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT) tie
    50%, 95% interval 29–71%
    10Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)
    39%, 95% interval 20–61%
      Show the numbers
      ModelScore95% CI
      DeepSeek V4.1 Flash high · Native FP8, FP4 experts89%67–97
      GLM-5.3 Flash high · EXL3 4.0bpw experts, FP8 dense83%61–94
      Gemma 4 31B thinking on · QAT W4A1661%39–80
      Qwen3.8 Flash-Next medium · NVFP461%39–80
      Nemotron 3.5 Super VL thinking on · NVFP4 mixed precision (NVIDIA ModelOpt)61%39–80
      Muse Glimmer 30B high · NVFP4 + FP8 mixed (NVIDIA ModelOpt)56%34–75
      MiMo V2.6 Flash thinking on · FP8 + MXFP4 experts56%34–75
      MiniCPM-V 4.7 35B-A3B thinking on · BF1650%29–71
      MiniMax M3.1 Flash high · NVFP4 experts, MXFP8 (QAT)50%29–71
      Ornith 1.5 122B thinking on · NVFP4 experts, FP8 attention (NVIDIA ModelOpt)39%20–61
      Nonprofit Success Pack: c@3 by configuration with 95% confidence intervals
      #Model and configScore c@3 · 95% CITasks
      1Gemini 3.8 Flash effort medium
      100%95% CI 82–100
      		18
      1Gemini 4 Argon effort medium
      100%95% CI 82–100
      		18
      1Grok 4.6 effort high
      100%95% CI 82–100
      		18
      1Muse Wonder effort low
      100%95% CI 82–100
      		18
      5Claude Fable 5.1 effort high
      94%95% CI 74–99
      		18
      5Claude Opus 5.5 effort medium
      94%95% CI 74–99
      		18
      5Claude Opus 5.5 effort low
      94%95% CI 74–99
      		18
      5Claude Sonnet 5.5 effort high
      94%95% CI 74–99
      		18
      5Claude Sonnet 5.5 effort low
      94%95% CI 74–99
      		18
      5Claude Sonnet 5.5 effort medium
      94%95% CI 74–99
      		18
      5Gemini 4 Argon effort low
      94%95% CI 74–99
      		18
      5Gemini 4 Argon effort high
      94%95% CI 74–99
      		18
      5GPT-5.6 Sol effort medium
      94%95% CI 74–99
      		18
      5GPT-6 Astra effort medium
      94%95% CI 74–99
      		18
      5Grok 4.7 effort high
      94%95% CI 74–99
      		18
      5Muse Wonder effort medium
      94%95% CI 74–99
      		18
      5Muse Wonder effort max
      94%95% CI 74–99
      		18
      5Muse Wonder effort high
      94%95% CI 74–99
      		18
      19Claude Opus 4.8 effort high
      89%95% CI 67–97
      		18
      19Claude Opus 4.8 effort medium
      89%95% CI 67–97
      		18
      19Claude Opus 4.8 effort low
      89%95% CI 67–97
      		18
      19Claude Opus 4.8 thinking off
      89%95% CI 67–97
      		18
      19Claude Opus 5.5 effort high
      89%95% CI 67–97
      		18
      19DeepSeek V4.1 Flash FP8 · effort high
      89%95% CI 67–97
      		18
      19DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high
      89%95% CI 67–97
      		18
      19GPT-6 Astra effort high
      89%95% CI 67–97
      		18
      27Claude Haiku 5.5 effort medium
      83%95% CI 61–94
      		18
      27Claude Sonnet 5 effort high
      83%95% CI 61–94
      		18
      27DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high
      83%95% CI 61–94
      		18
      27GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high
      83%95% CI 61–94
      		18
      27GPT-6 Luna effort medium
      83%95% CI 61–94
      		18
      32Claude Haiku 5.5 effort high
      78%95% CI 55–91
      		18
      32Claude Haiku 5.5 effort low
      78%95% CI 55–91
      		18
      32Claude Sonnet 5 effort low
      78%95% CI 55–91
      		18
      32GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high
      78%95% CI 55–91
      		18
      32GPT-6.1 Sol effort high
      78%95% CI 55–91
      		18
      32MiMo V2.6 Pro FP8 · thinking on
      78%95% CI 55–91
      		18
      38Claude Sonnet 5 thinking off
      72%95% CI 49–88
      		18
      38DeepSeek V4 Pro FP8 · effort high
      72%95% CI 49–88
      		18
      38DeepSeek V4.1 Flash 1017 Native FP8, FP4 experts · SGLang · effort high
      72%95% CI 49–88
      		18
      41Claude Sonnet 5 effort medium
      67%95% CI 44–84
      		18
      41GLM-5.3 Flash FP8 · effort high
      67%95% CI 44–84
      		18
      41Mistral Large 4 effort high
      67%95% CI 44–84
      		18
      44Gemma 4 26B-A4B NVFP4 · vLLM · thinking on
      61%95% CI 39–80
      		18
      44Gemma 4 31B QAT W4A16 · vLLM · thinking on
      61%95% CI 39–80
      		18
      44GLM-5.3 FP8 · effort low
      61%95% CI 39–80
      		18
      44GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low
      61%95% CI 39–80
      		18
      44GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off
      61%95% CI 39–80
      		18
      44Nemotron 3.5 Super VL NVFP4 mixed precision (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      61%95% CI 39–80
      		18
      44Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium
      61%95% CI 39–80
      		18
      44Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium
      61%95% CI 39–80
      		18
      44Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      61%95% CI 39–80
      		18
      53Claude Haiku 4.5 thinking on
      56%95% CI 34–75
      		18
      53Claude Haiku 4.5 thinking off
      56%95% CI 34–75
      		18
      53Inkling Small FP8 · effort high
      56%95% CI 34–75
      		18
      53MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on
      56%95% CI 34–75
      		18
      53MiMo V2.6 Flash FP8 · thinking on
      56%95% CI 34–75
      		18
      53Muse Glimmer 30B NVFP4 + FP8 mixed (NVIDIA ModelOpt) · vLLM · effort high
      56%95% CI 34–75
      		18
      53Nemotron 3 Ultra thinking on
      56%95% CI 34–75
      		18
      53Qwen3.8 27B MLX 8-bit · MTPLX · effort low
      56%95% CI 34–75
      		18
      53Qwen3.8 27B MLX 4-bit · MTPLX · effort low
      56%95% CI 34–75
      		18
      53Qwen3.8 27B Splash 4-bit · Splash · effort low
      56%95% CI 34–75
      		18
      53Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium
      56%95% CI 34–75
      		18
      64MiniCPM-V 4.7 35B-A3B BF16 · vLLM · thinking on
      50%95% CI 29–71
      		18
      64MiniCPM-V 4.7 35B-A3B INT4 W4A16 experts, BF16 dense · vLLM · thinking on
      50%95% CI 29–71
      		18
      64MiniMax M3.1 Flash NVFP4 experts, MXFP8 (QAT) · vLLM · effort high
      50%95% CI 29–71
      		18
      64Nemotron 3.5 Bolt IQ2_XXS + NVFP4 + Q8_0 mixed (NVIDIA ModelOpt) · TensorFold (Zig) · thinking on
      50%95% CI 29–71
      		18
      64Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on
      50%95% CI 29–71
      		18
      64Qwen3.8 27B AWQ-INT4 · vLLM · effort low
      50%95% CI 29–71
      		18
      64Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium
      50%95% CI 29–71
      		18
      71Qwen3.8 27B AWQ-INT4 · vLLM · effort medium
      47%95% CI 27–69
      		18
      72DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high
      44%95% CI 25–66
      		18
      72Ling 3.0 Flash BF16 · thinking on
      44%95% CI 25–66
      		18
      72Qwen3.8 Flash-Next thinking on
      44%95% CI 25–66
      		18
      75Laguna S 2.1 FP4 · thinking on
      39%95% CI 20–61
      		18
      75Nemotron 3.5 Lightning NVFP4 · thinking on
      39%95% CI 20–61
      		18
      75Ornith 1.5 122B NVFP4 experts, FP8 attention (NVIDIA ModelOpt) · vLLM · thinking on
      39%95% CI 20–61
      		18
      75Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh
      39%95% CI 20–61
      		18

      Tasks

      What's in the suite

      Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.

      Nonprofit Success Pack tasks with difficulty and solve rate
      TaskDifficultyFrom resultsSolve rate
      NPSP versus Nonprofit Cloudnpsp-vs-nonprofit-cloudeasyeasy
      100%
      Query pledge payments still duenpsp-soql-unpaid-payments-dueeasyeasy
      97%
      Record a donation the NPSP waynpsp-record-gifteasymedium
      39%
      Set donors' current employer with NPSP affiliationsnpsp-affiliation-primary-employereasyeasy
      68%
      Which NPSP object API names are real?npsp-namespace-objectseasymedium
      49%
      Annual-report totals per GAUnpsp-soql-gau-totalsmediumeasy
      77%
      Create one NPSP household for several new contactsnpsp-household-create-membersmediumhard
      25%
      Predict NPSP's household formal greetingnpsp-household-formal-greetingmediumhard
      32%
      Promote NPSP Customizable Rollups to productionnpsp-crlp-promote-to-productionmediumeasy
      98%
      Record parent relationships with NPSP reciprocalsnpsp-relationship-parent-childmediummedium
      62%
      Replace a pledge's open payments with an installment plannpsp-payment-installmentsmediummedium
      65%
      Skip an NPSP trigger handler for one integration usernpsp-trigger-handler-exclude-usermediumeasy
      100%
      Split a gift across GAUs by percentagenpsp-gau-percent-allocationsmediumeasy
      79%
      Bulk partial soft credits linked to contact rolesnpsp-partial-soft-creditshardhard
      25%
      Create monthly Enhanced Recurring Donationsnpsp-enhanced-rd-monthlyhardhard
      32%
      Import web gifts through NPSP Data Importer (BDI)npsp-bdi-gift-importhardhard
      5%
      Major gift follow-up as an NPSP TDTM handlernpsp-tdtm-major-gift-handlerhardhard
      32%
      Testing a custom TDTM handler in a subscriber orgnpsp-tdtm-test-registrationhardhard
      18%

      Private evaluations

      Want this for your team?

      Azul Labs runs Forcebench privately for two kinds of Salesforce team.

      Choosing a model to run yourself?

      A private benchmark of your shortlist

      Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

      • Private tasks written from your work only ever run on local models; they are never sent to an API.
      • Re-testing after any change of model, quantisation or engine is available.
      • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
      Get a private benchmarkHow it works

      Building or buying a Salesforce AI tool?

      A private evaluation of your tool

      Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

      • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
      • Repeat evaluations after your releases are available.
      • Typical turnaround is 2–3 weeks.
      Ask about an evaluationHow it works

      Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

      Or email leo@azl.au · azl.au