The skills mostly go unread

How it was run

The same lite tasks (4 per suite), graded by the same graders, with the same 32,768-token output budget for each reply. The single answers are the leaderboard's own runs of the same model, effort and quantisation. Every task was answered once in each arm; for Gemma 4 26B-A4B and Qwen3.8 27B, the 24 tasks of the six suites the skills cover most directly (Apex, the API, Flow, LWC, permission sets and SOQL) were answered three more times in each arm, the two arms side by side, and a task's score is the share of its answers that passed.

The agent is opencode 2.0.21, in an isolated container: the task's files in its working folder, a shell, no network and no Salesforce org. It gets the same message a single-answer run gets, plus one note that the files are in its folder and that it can edit them and run commands. Each task may use up to 60 model calls and an hour.

The skills are Salesforce's own, sf-skills 1.58.0: the 16 of its 240 skills on the suites' subjects (Apex and Apex tests, Lightning Web Components, Flow, SOQL, objects, fields and other metadata, permission sets, the API references, and deploying with the sf CLI), chosen by subject, never by task. The agent sees each skill's name and description and loads a skill when it decides to. The full method, the exact commands and every run behind these numbers are in docs/agent-track.md.

Did the agent use the skills?

Rarely. The agent can use a skill two ways: load it with its skill tool, or read and search the skill's files (instructions, reference documents, sample classes) directly. DeepSeek V4.1 Flash used the skills on 15 of its 60 tasks, Qwen3.8 27B in 20 of its 132 answers, and Gemma 4 26B-A4B in 1 of 132, though the list of skills was in front of every one of them, every time. With so few answers that used a skill, they say little about what using one did.

One pattern looked like the skills doing harm and turned out not to be. Every Qwen3.8 27B answer that loaded the permission set skill on its own failed to deploy (0 of 7): Qwen had added fields a permission set does not have. But Qwen does that in every arm, without the skills too, and with the skill always loaded it passed 4 of 8. These counts come from the agents' event logs, which stay with the raw replies and are not published; how they are counted is in the benchmark's documentation.

With the skills always loaded

Left to itself, the agent rarely loads a skill. To take that choice away, a third arm put the skill on each task's subject in front of the task, exactly as the agent's own skill tool would hand it over, for the six suites with one clear match: Apex, LWC, Flow, SOQL, permissions and the API. On those tasks, with the same model's single answers for comparison (the interval is for the change from no skills, resampling the tasks):

DeepSeek V4.1 Flash: with the skills always loaded, against the agent without them, clearly lower (−21 pts, 95% CI −37 pts to −4 pts); better on 0 tasks and worse on 5. With one answer per task, its agent without skills scored 88% against 71% for its own single answers to the same tasks, clearly higher (+17 pts, 95% CI +4 pts to +33 pts). Against those single answers, the skills always loaded gave no clear difference (−4 pts, 95% CI −21 pts to +13 pts); this data can't rule out a change of up to 21 points.

Gemma 4 26B-A4B: with the skills always loaded, against the agent without them, clearly higher (+16 pts, 95% CI +3 pts to +29 pts); better on 9 tasks and worse on 2. Most of the change is in Apex (25% → 69% of answers passing) and permission sets (38% → 75% of answers passing).

Qwen3.8 27B: with the skills always loaded, against the agent without them, no clear difference (+3 pts, 95% CI −6 pts to +13 pts); this data can't rule out a change of up to 13 points; better on 5 tasks and worse on 2.

2 of the 9 comparisons on this page are clear on their own (DeepSeek V4.1 Flash, skills always loaded: clearly lower; Gemma 4 26B-A4B, skills always loaded: clearly higher); with this many comparisons across three models and several arms, one or two clear results could partly be chance. Where the skills helped, the change was largest where they are most concrete: the Apex skill brings sample classes for each kind of job, and the permission set skill spells out the metadata. One possible explanation for why only some models gained, which this study doesn't test: a model that never reaches for the skills on its own can still use them once they are in front of it, while others may already know what the skills teach.

What the agent costs

An agent works over many turns and re-reads its whole conversation on each one, so it takes longer per task than a single answer. The times aren't compared here: the single answers and the agent runs ran under different conditions. The skills add input on top: their descriptions ride along on every request, and DeepSeek V4.1 Flash read about twice as many input tokens with them as without. A few agent sessions ended with no answer at all: the model's first turn spent the whole output budget thinking, before it read a file.

Why the agent alone changed little

These tasks give the model everything it needs in one message: the requirement, the files to change and the format of the answer. What an agent adds is a loop of trying, checking and fixing. Without an org, the agent cannot deploy, run Apex tests or query data, so it can only re-read its own work, not check it against Salesforce. And the skills only help a model that reads them, which, left to choose, these models rarely did.

What this does not show

Three open-weight local models and no API ones, on 60 tasks, mostly one answer per task (four on 24 of them for two models): a change smaller than about 9 points could go undetected. The always-loaded arm ran for three models and six suites only. One agent: others may behave differently; here, moving into the agent made no clear difference for any of the models. And the skills were measured as guidance only. Salesforce built them for an agent working against a real org and its own tools, where they deploy, run tests and scan code (the Flow skill, for one, builds flows through a Salesforce MCP tool); without an org or those tools, that part cannot run, and that loop is not what this study measures.

What to take from it

For the work Forcebench measures, the model matters most. A coding agent is a good way to work, with your files and your org at hand, but here, without an org, it made no clear difference to the same model's Salesforce code. Loaded on every task, the skills scored clearly higher for 1 and clearly lower for 1 of 3 models; left to the agent, they were rarely loaded, so do not count on the agent to reach for them. If your agent lets you load a skill for the task at hand, do. Pick the model on the leaderboard, then work with it however suits you.