An MRI of a model doing Salesforce work

How it was looked at

The model is Qwen3.8 27B in MLX 8-bit, the same weights Forcebench grades as qwen3.8-27b-mlx-8bit. It ran locally with its internals exposed: after each of its 64 layers, the activations (the residual stream) can be read, swapped for another run’s or nudged in a direction. No new answers were graded: study 3 uses the grades Forcebench already has for this model. Reasoning was off unless a section says otherwise.

• The logit lens reads what the model would say next if it stopped at a given layer.
• Linear probes test whether something can be read off the activations at a layer, on examples held out of their training.
• Activation patching swaps one run’s activations into another to see where a decision is made.
• Steering adds a direction to the activations to check that it does what a probe says it means.

01 · Apex and Java

Does it think in Java while writing Apex?

Apex looks like Java, and there is far more Java to learn from, so a model could plausibly draft in Java and translate. To test that, 64 short coding requests were each written three ways: asking for Apex, asking for Java, and naming no language. Each stops at a point where the two languages part ways, such as System. before debug( or out.println(, across 10 kinds of idiom. At that point the logit lens reads, at every layer, which language’s next token is ahead.

Asked for Apex, the model ends with the Apex token ahead in 60 of 64 pairs. On the way there, from layer 49 to 55, the Java token is ahead, most at layer 53 (−2.5), where it leads in 75% of pairs. Apex takes over at layer 57 and finishes far ahead.

That dip is not the model translating from Java as such. Asked for no language at all, it reads like the Java request to the end, and it picks the Java form in 92% of pairs: Java is its default for this kind of code. Up to layer 50, the Apex request follows that default closely, pair by pair: a correlation of at least 0.81 from layer 24, and never more than 0.8 log-odds apart in the median. From layer 49 the default swings hard towards the Java token, and the Apex correction follows a few layers behind; in between, Java is ahead. Where the default is Apex’s own form, string equality with == (5 pairs), the middle layers favour == even when asked for Java. So the middle of the network drafts the most likely code with little regard for the language asked for, and the language is applied late.

The model does know the language early. A linear probe on the same position tells the Apex request from the Java one with at least 96% accuracy from layer 1 on, on pairs it wasn’t trained on. It just doesn’t use that for the next token until about layer 50.

Patching finds where the switch happens. Writing the Java request’s activations into the Apex request’s run at one layer, at the last position only, moves 5% of the answer at layer 32 and 35% at layer 48, then 76% at layer 52 and 89% at layer 56. Before about layer 50 the language is still being read in from the request by later layers; between layers 50 and 54 the decision settles at the position that writes the next token. The other way round, Apex into Java, the last position moves 8% at layer 48 and 80% at layer 52. Swapping all of the code that way gives uneven results in the middle layers, with wide intervals, and isn’t read into here.

Steering checks that this is one direction the model uses. The average difference between the Apex and Java requests at layer 50, at the last position, works like a switch. Taken away from the Apex request, it cuts the pairs where the Apex form wins from 60 of 64 to 21; a random direction of the same length leaves 59. Added to the Java request, it gives the Apex form in 15 of 64 pairs, up from 3 (random: 4). Each test leaves the pair it is tested on out of the direction.

That direction, added or taken away at every position at layer 50, was then tried on 12 Apex requests the model answered in full: tests, utility methods and a trigger handler. The replies were searched for things an Apex compiler rejects or only Java writes, such as System.out, JUnit’s @Test, int, double-quoted strings, ArrayList and import. Unsteered, there were 0. With the direction taken away, the model answered in Java: 10 of 12 replies opened a Java code block, and all 12 had Java-isms, 98 in all. A random direction of the same length left 0.

With the direction taken away, 53 of the 98 are double-quoted strings and 19 are JUnit’s @Test.

The opening lines of one reply, to “Write an Apex unit test for a DiscountCalculator.apply(Decimal amount) method that gives 10% off amounts over 1,000. Reply with just the code.”:

@isTest
private class DiscountCalculatorTest {
    @isTest
    static void testApplyDiscountForAmountOver1000() {
        Decimal amount = 1500.00;
        Decimal expected = 1350.00;
        Decimal actual = DiscountCalculator.apply(amount);
        System.assertEquals(expected, actual, 'Discount should be 10% off for amounts over 1000');
    }

public class DiscountCalculatorTest {
    @Test
    public void testApplyDiscount() {
        // Test case: Amount over 1000 should get 10% discount
        double amount = 1500.0;
        double expected = amount * 0.9;
        double result = DiscountCalculator.apply(amount);
        assertEquals(expected, result, "Discount should be 10% for amounts over 1000");

Some Java does reach the answers. In 4 of the 6 assertion pairs, the model’s own next token when asked for Apex is a Java library’s (.assertEquals, .assertFalse, .notNull), which Apex’s Assert class doesn’t have. By kind of idiom:

Final answers are the median log-odds at the last layer: above zero the Apex form wins.

02 · Clicks or code

Does it reach for code when clicks would do?

Much Salesforce work can be done two ways: with clicks, the platform’s declarative features such as validation rules, formula fields and Flow, or with code, which on Salesforce means Apex. Salesforce’s guidance is to use clicks where they do the job and code where they don’t. A model that has read far more code than configuration might reach for code anyway.

The study wrote 36 requirements, each with a textbook answer from that guidance: 12 that a standard feature does (a validation rule, a formula or roll-up summary field, an assignment, escalation, duplicate or sharing rule, an approval process), 12 that a flow does, and 12 that need Apex (high volumes, logic across many objects, callouts with retries, a custom endpoint, parsing a file). With reasoning off, the model was asked about each one three ways: how it would build it, in one sentence; the one feature it would use; and, offered both, clicks or code in one word, asked in both orders with the chance of each answer read directly. An answer reaches for code if it names custom code: Apex, a Lightning Web Component, or a flow that calls Apex.

Where clicks will do, it almost never reaches for code. Its one-sentence answer names code for 1 of the 24 requirements that a standard feature or a flow can do, and offered both, it gives “code” a 2% chance on average. Where Apex is needed, its one-sentence answer names code for 11 of 12, 3 of them as a flow that calls Apex, which the guidance also allows, and offered both, it gives “code” 89%.

Asked for a single feature, it names code less often, even where code is needed: 3 of 12 times. For those requirements it most often names Platform Events (4 of 12), which carries events to a flow or to Apex rather than doing the work itself.

Asked for the one Salesforce feature it would use, with nobody named: its answers as it wrote them.

Who asks moves the open answer, one way. When a developer asks, its one-sentence answer reaches for code for 4 of the 12 requirements a flow can do, against 1 with nobody named and 0 when an admin asks: +33 points against the admin (95% interval +8 to +58), and not clearly different from nobody named (+25, 0 to +50). Requirements a standard feature does don’t move. Offered a straight choice, the asker barely matters: on requirements that need Apex, an admin asking lowers the chance of “code” by 6 points (95% interval 3 to 9), and a developer asking is not clearly different from nobody named (+1, 0 to +2).

Its answers also show their age. With nobody named, 33% of its one-sentence answers for requirements clicks can do suggest Process Builder or Workflow Rules, often as “a Flow (or Process Builder)”. Salesforce has retired both in favour of Flow.

Inside, there is no hidden pull towards code that the model then overrules. Read through the logit lens, the one-word choice looks much the same for all three kinds of requirement up to layer 46, never more than 0.53 log-odds apart (at layer 36): the middle layers’ larger swings are shared by every requirement alike. The one steady difference is a slight lean in the textbook order from layer 30 to 37, with the requirements that need Apex a little nearer code than the flow ones, and those a little nearer than the standard ones. From layer 47 the requirements that need Apex pull ahead, by several log-odds from layer 55, and the others settle firmly on clicks only from layer 58. The choice is made late, like the Apex form in study 1.

03 · The task at hand

Does it know it’ll fail?

Forcebench graded this model, in these weights, on the 272 public tasks of benchmark 0.1.0, with reasoning effort low: it passed 78 (29%; the leaderboard’s suite-averaged score is 28%). Each task was shown to it again exactly as in that run, and its activations were read at the end of the prompt, before it had started reasoning. Separately, with reasoning off, it was shown each task with a question added at the end that told it not to solve the task yet: how likely its answer would be to pass, answered with one digit from 0 to 9.

Each signal is scored by its AUROC: the chance that a task it passed ranks above a task it failed, where 0.5 is a coin toss and 1 is perfect.

Before it has written a word, a linear probe on its activations at layer 40, a layer fixed before looking, tells the tasks it will pass from the ones it will fail with an AUROC of 0.81 (95% interval 0.76–0.86); the best layer, 46, reaches 0.83. With the labels shuffled, the best of all 64 layers gets past 0.62 only one time in twenty.

Some of that is just the kind of task. The suite, the difficulty and the prompt’s length alone give 0.75. The probe adds to that (+0.07, 95% interval +0.03 to +0.10), and it still reaches 0.70 comparing tasks within the same suite, where the baseline manages 0.59. Reading its own finished answer doesn’t help: a probe at the end of the answer does about as well (0.78 at layer 40).

Each label is one graded attempt at temperature 1.0, so some passes and fails are luck. As a check, 6 other runs of the same model in AWQ int4, at low, medium and xhigh effort, give each task a pass rate over at least 4 attempts. That pass rate predicts this run’s results with an AUROC of 0.89, about as well as anything could, and a probe trained on that pass rate predicts it with a rank correlation of 0.59 (the suite baseline: 0.40).

What it says is another matter. Asked how likely its answer is to pass, it gives 7.6 out of 9 on average (7.9 for tasks it went on to pass, 7.5 for ones it failed), against a pass rate of 29%. Its stated confidence separates passes from fails with an AUROC of 0.64, below the suite baseline (−0.12, 95% interval −0.19 to −0.04); the probe beats it by +0.18 (+0.10 to +0.26). Only its most confident sixth of tasks pass noticeably more often.

Two things tilt that comparison against what it says. It answered with reasoning off, while the graded answers were written with reasoning on. And the probe, like the baseline, was fitted to these grades (scored only on tasks held out of the fit), while its stated confidence was never fitted to anything. So the fair yardstick for the probe is the baseline. What stands either way: the model rates its chances far above how often it passes, and its rating tells passes from fails less well than the kind of task does.

Graded run: 20260927T004832Z_qwen3.8-27b-mlx-8bit@low. The stated chance is the expected digit from the model’s probabilities for 0 to 9, asked as “Don't solve it yet. How likely is it that your answer would pass the grader's hidden tests? Reply with one digit, 0 (certain to fail) to 9 (certain to pass).”

What this doesn’t show

• It’s one model, in one quantisation. Another model, or this one at full precision, may draw its lines elsewhere.
• The code pairs and requirements were written for this study, and the textbook answers are its reading of Salesforce’s guidance: a few have more than one fair answer. They are small samples: 64 code pairs, and 12, 12 and 12 requirements. Study 3 uses the 272 public tasks of benchmark 0.1.0, the version its graded run used.
• Each open answer in study 2 is one reply with reasoning off, the model’s most likely words, sorted by the features it names. With reasoning on, or sampled, it may answer differently.
• The logit lens is approximate in the middle layers, which weren’t trained to be read this way. The probe and patching results don’t depend on it.
• The pass or fail of each task is one attempt, and a probe that predicts it is a correlation: it shows the information is there before the model starts, not that the model uses it.
• The after-answer read sees the model’s final answer but not the reasoning before it, which the study left out.

Numbers from the study file generated 22 October 2026. Code, stimuli and per-item results: github.com/forcebench-ai/forcebench-mri.