affordance

Showcase  /  Built with Affordance

We trained Qwen3-4B to work in a plain document instead of a chat, from 11% to 78% right

We gave Qwen3-4B, a chat model, a task written as a plain document instead of a chat. At first it answered 10.9% of attempts correctly. After 70 steps of training it answered 77.7%, the same as the model trained in its usual chat format.

Model

Qwen3-4B

Right answers, at first

10.9%

After 70 steps

77.7%

A language model reads and writes tokens, which are numbers that each stand for a short piece of text. A chat model is trained on conversations laid out one exact way, with tokens that mark who is speaking. The program that lays them out is called a chat template. Our document has no speakers and does not use the chat template.

We trained with reinforcement learning. The model tries the task many times, and the attempts that got the right answer are made more likely. Affordance is an open-source program that builds the exact tokens a model reads from an app, a file written in TSX, the language of React web pages. We wrote the document layout as a 24-line Affordance app.

The task and the document

The model sees a notebook of 16 to 48 four-digit numbers and has to submit the largest. It can only see about 40 tokens of the notebook at a time, a few rows. It has three tools: scroll the notebook, write itself a note, and submit an answer.

The document starts with the task and the list of tools. Then each numbered step shows the notebook, then Action: and the model's reply, then Result:. This is the start of one attempt before training, with the model's own tokens highlighted:

what the model reads and writes, before training
Task: The notebook below has 21 numbers, one per row, but you can only see a few rows at a time. Find the largest number in the notebook and submit it. Use scroll to move through the notebook and note to keep track of what you have seen.

Tools. Take one step at a time by calling one tool, written as
<tool_call>
{"name": <tool name>, "arguments": {<arguments>}}
</tool_call><|im_end|>
- scroll: Scroll the notebook by a number of rows (negative scrolls up). Arguments: {"by":{"type":"number"}}
- note: Replace your notes with new text. Arguments: {"text":{"type":"string"}}
- submit: Submit your final answer: the largest number in the notebook. This ends the task. Arguments: {"answer":{"type":"number"}}

Step 1
Notebook (21 rows), starting at row 0:
0: 8857
1: 7298
2: 8991
3:...
Your notes: (empty)
Action:
<tool_call>
{"name": "scroll", "arguments": {"by": 1}}
</tool_call><|im_end|>
Result: The notebook now starts at row 1.

Step 2
Notebook (21 rows), starting at row 1:
1: 7298
2: 8991
3: 3598
4:...
Your notes: (empty)
Action:
<tool_call>
{"name": "scroll", "arguments": {"by": 1}}
</tool_call><|im_end|>
Result: The notebook now starts at row 2.

Step 3
Notebook (21 rows), starting at row 2:
2: 8991
3: 3598
4: 6564
5:...
Your notes: (empty)
Action:
<tool_call>
{"name": "scroll", "arguments": {"by": 1}}
</tool_call><|im_end|>
Result: The notebook now starts at row 3.

13 more steps: the model scrolls by 1 each time until it runs out of steps

The model writes its tool calls in the form it learned in chat, and ends each step with <|im_end|>, the special token that ends its turn in a chat. The tool list shows both in its example. Our first version left <|im_end|> out of the example, and the model kept writing past its own step in 82% of replies. With the token in the example, 96% of its replies were well-formed tool calls.

Here is the app that lays out the document:

doc_app.tsxlines 8 to 23
export function DocApp({ children }) {
  const input = templateInput(children, null, true);
  const fns = input.tools.map(t => t.function);
  let step = 0;
  const body = input.turns.map((m, i) => {
    const content = typeof m.content === "string" || isFragment(m.content) ? m.content : "";
    if (i === 0) return ["Task: ", content, "\n\nTools. Take one step at a time by calling one tool, written as\n", tok.tool_call,
      "\n{\"name\": <tool name>, \"arguments\": {<arguments>}}\n", tok.tool_call_end, tok.im_end, "\n",
      fns.map(f => `- ${f.name}: ${f.description} Arguments: ${JSON.stringify(f.parameters.properties)}\n`).join("")];
    if (m.role === "user") { step += 1; return [`\nStep ${step}\n`, content]; }
    if (m.role === "assistant") return ["Action:\n", <turn role="assistant">{tokens(m.ids)}</turn>, "\n"];
    if (m.role === "tool") return ["Result: ", content, "\n"];
    return "";
  });
  return [<>{body}{"Action:\n"}</>,
          <span label="protocol" syntax="hermes" stop={[tok.im_end.id, tok.endoftext.id]} in_think={false}/>];

The model's replies go back into the document as tokens(m.ids), the numbers of the tokens it wrote, unchanged.

Before training

In the document, the base model scrolled one row at a time and rarely answered. It submitted an answer in 24% of attempts, against 95% in its chat format. It answered 10.9% correctly, against 40.2% in chat.

Accuracy during training

Share of held-out attempts answered correctly, by training step

The same 256 notebooks, never trained on, every 10 steps. Same model, task and training settings in both runs.

Plain document, no chat templateChat format
0%25%50%75%100%010203040506070training stepPlain document, no chat template, step 0: 10.9%Plain document, no chat template, step 10: 62.1%Plain document, no chat template, step 20: 73.4%Plain document, no chat template, step 30: 73.0%Plain document, no chat template, step 40: 75.4%Plain document, no chat template, step 50: 77.0%Plain document, no chat template, step 60: 77.0%Plain document, no chat template, step 70: 77.7%Chat format, step 0: 40.2%Chat format, step 10: 48.4%Chat format, step 20: 56.3%Chat format, step 30: 75.8%Chat format, step 40: 77.3%Chat format, step 50: 77.3%Chat format, step 60: 77.3%Chat format, step 70: 77.3%document 78%chat 77%

The document run went from 10.9% to 62.1% in the first ten steps, and caught up with the chat run by step 50.

Held-out attempts in the documentBefore trainingAfter 70 steps
Attempts that ended with an answer23.8%100.0%
Attempts that scrolled to the last row26.2%87.1%
Attempts answered correctly10.9%77.7%
Tokens the model wrote per attempt489404

An attempt after training

The trained model scrolls 3 rows at a time, writes every number it has seen into its notes, and submits. This attempt is from the final test:

after 70 steps, a 20-row notebook, answered correctly
the task, the tools, and steps 1 to 4, each scrolling by 3
Step 5
Notebook (20 rows), starting at row 12:
12: 4978
13: 6864
14: 231...
Your notes: (empty)
Action:
<tool_call>
{"name": "scroll", "arguments": {"by": 3}}
</tool_call><|im_end|>
Result: The notebook now starts at row 15.

Step 6
Notebook (20 rows), starting at row 15:
15: 4442
16: 9334
17: 708...
Your notes: (empty)
Action:
<tool_call>
{"name": "scroll", "arguments": {"by": 3}}
</tool_call><|im_end|>
Result: The notebook now starts at row 18.

Step 7
Notebook (20 rows), starting at row 18:
18: 6706
19: 5961

Your notes: (empty)
Action:
<tool_call>
{"name": "note", "arguments": {"text": "Collected numbers: 7014, 6124, 8299, 8775, 8796, 7926, 8054, 3523, 3292, 2168, 7515, 2886, 4978, 6864, 231, 4442, 9334, 708"}}
</tool_call><|im_end|>
Result: Noted.

Step 8
Notebook (20 rows), starting at row 18:
18: 6706
19: 5961

Your notes: Collected numbers: 7014, 6124, 8299, 8775, 8796, 7926, 8054, 3523, 3292, 2168, 7515, 2886, 4978, 6864, 231, 4442, 9334, 708
Action:
<tool_call>
{"name": "submit", "arguments": {"answer": 9334}}
</tool_call><|im_end|>

Its notes include 231 and 708. Those are rows 14 and 17, which the notebook window cut off after three digits, and the model copied the digits it could see.

Why the trained model still misses some

The 40-token window shows three full rows at the top of the notebook. From row 10 on, the row numbers have two digits and take more tokens, so only two rows fit and the third is cut off. The model learned to scroll by 3 in the first rows and kept doing it, so after row 10 it never sees every third number in full.

Of the 57 attempts it got wrong after training, 54 were notebooks whose largest number it never saw in full. The model trained in chat misses for the same reason: 58 of its 58 wrong answers.

Comparison with the server's records

We recorded every request to SGLang, the model server, and every reply. Then we compared the tokens Affordance exported for training with those records.

Training and testingCount
Attempts19,968
Model calls236,207
Prompt tokens180,963,409
Tokens the model wrote8,426,505
Tokens that differed from the server's records0

One training run on one task, with a 4-billion-parameter model.