Showcase / Benchmark
We trained Qwen3-4B on chat-template rebuilds of its replies, and it stopped writing the blank line before its tool calls
We trained the same model on the same task twice. In one run the trainer used the tokens the model actually wrote. In the other it used the usual rebuild of those replies from saved messages. After 70 steps, the second model had stopped writing a blank line before its tool calls.
Model
Qwen3-4BTraining steps
70Replies the rebuild changed
11.0%A language model reads and writes tokens, which are numbers that each stand for a short piece of text. A trainer adjusts the model so that the tokens of good attempts become more likely. To do that it needs the exact tokens the model wrote. If it is given different tokens, it trains the model toward something the model did not write.
How the usual setup rebuilds replies
An agent is a model calling tools, such as scroll or submit, in a loop. The program around it saves the conversation as a list of messages. When it trains, it turns those messages back into tokens with the model's chat template, the small program that lays out a conversation in the model's format.
The chat template writes each tool call its own way. Our model, Qwen3-4B, had written a sentence, a space and a blank line, then its tool call. The template wrote one newline:
will now submit this as the largest number. \n\n
<tool_call>
{"name": " will now submit this as the largest number.\n
<tool_call>
{"name": "noteThe rebuild has one token fewer, and one token the model never chose. In the first training step the rebuild changed 208 of 1,886 replies, 11.0%. We recorded every request and reply, so we could count each change.
Changed replies, by training step
A nat is a unit for how much a probability changed: 18.8 nats is a factor of about 150 million. The middle row is the largest change we saw in a single token's probability, between what the server sampled and what the trainer was given.
| Training step | 0 | 10 | 20 | 30 | 50 | 70 |
|---|---|---|---|---|---|---|
| Replies the rebuild changed | 11.0% | 1.8% | 1.5% | 0.3% | 0.1% | 0.0% |
| Largest change in one token's probability, nats | 18.8 | 18.3 | 7.7 | 0.4 | 0.4 | 0.7 |
| Affordance's tokens that differed from what the server processed | 0 | 0 | 0 | 0 | 0 | 0 |
The changed replies fell to zero because the model stopped writing the blank line the template removes.
Blank lines before the tool call, by training step
Share of replies with a blank line between the sentence and the tool call
Measured on the tokens each model wrote, every 10 training steps.
Trained on its own tokens, the model wrote the blank line in 82% of these replies at the start and 99% after 70 steps. A second run started at 89% and ended at 81%. Trained on rebuilds, the model started at 52% and reached 0% by step 50. It started lower because its own earlier replies came back to it in the template's form, and it copied them.
Effect on the server's cache
A model server keeps the work it did on the start of a prompt, so a new prompt that starts the same way is served faster. That stored work is the cache. When a rebuilt reply goes into the next prompt, the prompt stops matching the cache at the first changed token, and the server computes the rest again.
Before training, 19 of the standard setup's model calls had most of their prompt computed again, each one right after a rebuilt reply. With Affordance, none did. SGLang, the model server we used, served 84.1% of the standard setup's prompt tokens from its cache, and 89.4% of Affordance's.
One task with a 4-billion-parameter model.