affordance

Showcase  /  Benchmark

We trained Qwen3-4B on chat-template rebuilds of its replies, and it stopped writing the blank line before its tool calls

We trained the same model on the same task twice. In one run the trainer used the tokens the model actually wrote. In the other it used the usual rebuild of those replies from saved messages. After 70 steps, the second model had stopped writing a blank line before its tool calls.

Model

Qwen3-4B

Training steps

70

Replies the rebuild changed

11.0%

A language model reads and writes tokens, which are numbers that each stand for a short piece of text. A trainer adjusts the model so that the tokens of good attempts become more likely. To do that it needs the exact tokens the model wrote. If it is given different tokens, it trains the model toward something the model did not write.

How the usual setup rebuilds replies

An agent is a model calling tools, such as scroll or submit, in a loop. The program around it saves the conversation as a list of messages. When it trains, it turns those messages back into tokens with the model's chat template, the small program that lays out a conversation in the model's format.

The chat template writes each tool call its own way. Our model, Qwen3-4B, had written a sentence, a space and a blank line, then its tool call. The template wrote one newline:

what the model wrote
 will now submit this as the largest number. \n\n

<tool_call>
{"name": "
what the template rebuilds
 will now submit this as the largest number.\n
<tool_call>
{"name": "note
model's tokens. \n\n<tool_call>
rebuilt tokens.\n<tool_call>

The rebuild has one token fewer, and one token the model never chose. In the first training step the rebuild changed 208 of 1,886 replies, 11.0%. We recorded every request and reply, so we could count each change.

Changed replies, by training step

A nat is a unit for how much a probability changed: 18.8 nats is a factor of about 150 million. The middle row is the largest change we saw in a single token's probability, between what the server sampled and what the trainer was given.

Training step01020305070
Replies the rebuild changed11.0%1.8%1.5%0.3%0.1%0.0%
Largest change in one token's probability, nats18.818.37.70.40.40.7
Affordance's tokens that differed from what the server processed000000

The changed replies fell to zero because the model stopped writing the blank line the template removes.

Blank lines before the tool call, by training step

Share of replies with a blank line between the sentence and the tool call

Measured on the tokens each model wrote, every 10 training steps.

Trained on Affordance's exact tokens, first runTrained on Affordance's exact tokens, second runTrained on chat-template rebuilds
0%25%50%75%100%010203040506070training stepTrained on Affordance's exact tokens, first run, step 0: 82.1%Trained on Affordance's exact tokens, first run, step 10: 88.2%Trained on Affordance's exact tokens, first run, step 20: 97.4%Trained on Affordance's exact tokens, first run, step 30: 91.9%Trained on Affordance's exact tokens, first run, step 40: 91.8%Trained on Affordance's exact tokens, first run, step 50: 92.5%Trained on Affordance's exact tokens, first run, step 60: 96.1%Trained on Affordance's exact tokens, first run, step 70: 98.8%Trained on Affordance's exact tokens, second run, step 0: 88.8%Trained on Affordance's exact tokens, second run, step 10: 89.7%Trained on Affordance's exact tokens, second run, step 20: 64.9%Trained on Affordance's exact tokens, second run, step 30: 68.7%Trained on Affordance's exact tokens, second run, step 40: 78.6%Trained on Affordance's exact tokens, second run, step 50: 88.3%Trained on Affordance's exact tokens, second run, step 60: 89.6%Trained on Affordance's exact tokens, second run, step 70: 81.2%Trained on chat-template rebuilds, step 0: 51.5%Trained on chat-template rebuilds, step 10: 11.0%Trained on chat-template rebuilds, step 20: 16.8%Trained on chat-template rebuilds, step 30: 3.1%Trained on chat-template rebuilds, step 40: 6.1%Trained on chat-template rebuilds, step 50: 0.4%Trained on chat-template rebuilds, step 60: 0.4%Trained on chat-template rebuilds, step 70: 0.0%exact, run 1 99%exact, run 2 81%rebuilt 0%

Trained on its own tokens, the model wrote the blank line in 82% of these replies at the start and 99% after 70 steps. A second run started at 89% and ended at 81%. Trained on rebuilds, the model started at 52% and reached 0% by step 50. It started lower because its own earlier replies came back to it in the template's form, and it copied them.

Effect on the server's cache

A model server keeps the work it did on the start of a prompt, so a new prompt that starts the same way is served faster. That stored work is the cache. When a rebuilt reply goes into the next prompt, the prompt stops matching the cache at the first changed token, and the server computes the rest again.

Before training, 19 of the standard setup's model calls had most of their prompt computed again, each one right after a rebuilt reply. With Affordance, none did. SGLang, the model server we used, served 84.1% of the standard setup's prompt tokens from its cache, and 89.4% of Affordance's.

One task with a 4-billion-parameter model.