Models / minicpm-v-4-7-35b-a3b

MiniCPM-V 4.7 35B-A3B

OpenBMB's model, with open weights. Forcebench has run it in 2 configurations on the Local board. API is models reached through a hosted service; Local, models served on Forcebench’s own hardware.

On the Local board

Models served on Forcebench’s own hardware. MiniCPM-V 4.7 35B-A3B is ranked by the highest-scoring of its 2 configurations: the 15th-highest score of 16 models, not clearly different from 5 other models. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board →

31%
within 3 attempts (c@3), 95% CI 26–36 · thinking on · 15th-highest score of 16 models, not clearly different from 5 other models
1.9k
output tokens per task · median 1.5k, p90 3.9k
219 s
median per task, every attempt counted, on 1× NVIDIA DGX Spark (GB10, 128 GB) · 90th percentile 806 s
1%
no answer within the 32,768-token budget
Every configuration
Within 3 attempts (c@3) with 95% intervals, highest first. Bars whose intervals overlap are not clearly different.
  • 95% interval
thinking on BF16 · vLLM · 1× NVIDIA DGX Spark (GB10, 128 GB)highest score
31%, 95% interval 26–36%
thinking on INT4 W4A16 experts, BF16 dense · vLLM · 1× NVIDIA GeForce RTX 5090 (32 GB)
28%, 95% interval 23–33%
    Show the numbers
    Every configuration of MiniCPM-V 4.7 35B-A3B on the Local board
    ConfigurationRank all configsWithin 3 attempts c@3 · 95% CIAgainst the highest by 95% interval overlapTokens mean · median · p90No answerLatency median per task, every attempt counted
    BF16 · vLLM · thinking on · 1× NVIDIA DGX Spark (GB10, 128 GB)highest score28
    31%95% CI 26–36
    		—1.9k1.5k · 3.9k1%219 s
    INT4 W4A16 experts, BF16 dense · vLLM · thinking on · 1× NVIDIA GeForce RTX 5090 (32 GB)29
    28%95% CI 23–33
    		not clearly different2.4k1.4k · 4.4k2%22 s

    Latency depends on the hardware: compare it only between configurations on the same hardware.

    Suites, at its highest-scoring configuration
    thinking on · BF16 · vLLM · 1× NVIDIA DGX Spark (GB10, 128 GB): within 3 attempts (c@3) on each suite with 95% intervals, against the median of the Local board's models, each at its highest-scoring configuration.
    • 95% interval
    • Local board median
    Apex
    50%, 95% interval 30–70%
    Salesforce APIs
    33%, 95% interval 16–56%
    CI/CD
    35%, 95% interval 17–59%
    Salesforce CLI
    20%, 95% interval 8–42%
    Salesforce docs
    18%, 95% interval 6–41%
    fflib Enterprise Patterns
    6%, 95% interval 1–26%
    Flow
    11%, 95% interval 3–33%
    Governor limits & pushback
    53%, 95% interval 32–73%
    Lightning Web Components
    44%, 95% interval 25–66%
    Nonprofit Success Pack
    50%, 95% interval 29–71%
    Packaging
    17%, 95% interval 6–39%
    Permissions & access
    33%, 95% interval 15–58%
    Scratch org definitions
    11%, 95% interval 3–33%
    SOQL
    65%, 95% interval 43–82%
    Trigger Actions Framework
    22%, 95% interval 9–45%
      Show the numbers
      Per-suite scores of MiniCPM-V 4.7 35B-A3B's highest-scoring configuration on the Local board, with the board's median
      Suitec@3 95% CILocal medianTasks
      Apex
      50%95% CI 30–70
      		70%20
      Salesforce APIs
      33%95% CI 16–56
      		42%18
      CI/CD
      35%95% CI 17–59
      		44%17
      Salesforce CLI
      20%95% CI 8–42
      		38%20
      Salesforce docs
      18%95% CI 6–41
      		12%17
      fflib Enterprise Patterns
      6%95% CI 1–26
      		17%18
      Flow
      11%95% CI 3–33
      		17%18
      Governor limits & pushback
      53%95% CI 32–73
      		76%19
      Lightning Web Components
      44%95% CI 25–66
      		83%18
      Nonprofit Success Pack
      50%95% CI 29–71
      		56%18
      Packaging
      17%95% CI 6–39
      		28%18
      Permissions & access
      33%95% CI 15–58
      		57%15
      Scratch org definitions
      11%95% CI 3–33
      		11%18
      SOQL
      65%95% CI 43–82
      		85%20
      Trigger Actions Framework
      22%95% CI 9–45
      		42%18

      By quantisation

      The same effort in different quantisations, each against the highest-precision one on the same tasks (a paired 95% interval), on pass@1, the first answer alone. Where the engine changes too, a difference can't be put down to quantisation alone. The quantisation study has every model.

      Each build at thinking on
      Overall pass@1 with 95% intervals, labelled with quantisation, engine and hardware. Intervals that overlap are not clearly different.
      • 95% interval
      BF16 vLLM · 1× NVIDIA DGX Spark (GB10, 128 GB)reference
      20%, 95% interval 16–25%
      INT4 W4A16 experts, BF16 dense vLLM · 1× NVIDIA GeForce RTX 5090 (32 GB)
      17%, 95% interval 13–21%
        Show the numbers
        Buildpass@195% CITokens
        BF16 · vLLM · 1× NVIDIA DGX Spark (GB10, 128 GB)20%16–251.9k
        INT4 W4A16 experts, BF16 dense · vLLM · 1× NVIDIA GeForce RTX 5090 (32 GB)17%13–212.4k
        MiniCPM-V 4.7 35B-A3B at thinking on by quantisation
        Quantisation · engine · thinking onpass@1 95% CIAgainst BF16 same tasks, pairedTokens
        BF16 · vLLM · 1× NVIDIA DGX Spark (GB10, 128 GB)reference
        20%95% CI 16–25
        		—1.9k
        INT4 W4A16 experts, BF16 dense · vLLM · 1× NVIDIA GeForce RTX 5090 (32 GB)
        17%95% CI 13–21
        		clearly lower (−3 pts, 95% CI −7 pts to −0.1 pts)2.4k

        In the studies

        Private evaluations

        Want this for your team?

        Azul Labs runs Forcebench privately for two kinds of Salesforce team.

        Choosing a model to run yourself?

        A private benchmark of your shortlist

        Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

        • Private tasks written from your work only ever run on local models; they are never sent to an API.
        • Re-testing after any change of model, quantisation or engine is available.
        • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
        Get a private benchmarkHow it works

        Building or buying a Salesforce AI tool?

        A private evaluation of your tool

        Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

        • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
        • Repeat evaluations after your releases are available.
        • Typical turnaround is 2–3 weeks.
        Ask about an evaluationHow it works

        Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

        Or email leo@azl.au · azl.au