affordance

Showcase  /  Benchmark

Affordance inside SGLang made replies start 24% sooner on 120,000-token prompts

We put Affordance inside SGLang, an open-source model server, to build the model's prompts. On Qwen3.6-35B-A3B, with prompts over 120,000 tokens, the first token of each reply came after 552 to 562 ms with stock SGLang and 421 to 422 ms with Affordance, over three runs each.

Model, on SGLang

Qwen3.6-35B-A3B

Prompt

over 120,000 tokens

Stock SGLang

552 to 562 ms

With Affordance

421 to 422 ms

Change

24% sooner

A language model reads and writes tokens, which are numbers that each stand for a short piece of text. A model server is the program that runs a model and answers requests. Affordance is an open-source program that builds the exact tokens a model reads from an app, a file written in TSX, the language of React web pages.

The work a server does before the model starts

A request arrives as a list of messages. The server joins them into one long string with the model's chat template, a small program written in a template language called Jinja. Then a tokenizer splits that string into tokens.

Both steps start from the first character on every request. An agent sends its whole conversation again at every step, with one new turn at the end. So at step 100, the server joins and tokenizes all 99 earlier turns again.

The change we made

Affordance goes from the messages straight to tokens. It keeps the tokens of turns it has already built, so each request costs only its new turn. We put it into SGLang and into vLLM, another open-source server, in place of their template and tokenizer step. On 2,703 test requests for SGLang and 2,880 for vLLM, including ones with tools, each server produced the same tokens with Affordance as without it.

Three models on two servers

Each run is one conversation that grows by about 1,600 tokens a turn, to 164,000 tokens (117,000 for gpt-oss-20b). We timed how long each reply took to produce its first token. Each server ran three conversations it had never seen.

Then each server ran the same conversations a second time. A server stores the work it did on a prompt it has seen, so the model's own work is small the second time, and building the prompt is most of the wait.

Model and server, prompts over 120,000 tokens (60,000 to 120,000 for gpt-oss-20b)Stock server, three runsWith AffordanceChangeSecond time: stockSecond time: with Affordance
Qwen3.6-35B-A3B, SGLang552 to 562 ms421 to 422 ms24% sooner165 to 192 ms42 to 45 ms
Qwen3.6-27B, SGLang1,103 to 1,143 ms967 to 969 ms14% sooner187 to 222 ms66 to 67 ms
gpt-oss-20b, SGLang440 to 486 ms447 to 466 ms2% sooner157 to 172 ms152 to 166 ms
Qwen3.6-35B-A3B, vLLM477 to 509 ms472 to 476 ms1% sooner218 to 266 ms209 to 213 ms
Qwen3.6-27B, vLLM1,170 to 1,175 ms1,131 to 1,143 ms3% sooner354 to 359 ms330 to 331 ms
gpt-oss-20b, vLLM394 to 395 ms368 to 369 ms7% sooner88 to 89 ms57 to 59 ms

On SGLang the time saved on the longest prompts was about the same for both Qwen models, 135 to 165 ms. Qwen3.6-35B-A3B runs faster than the 27B model, so the same saving is a larger share of its wait. When that model saw a prompt the second time, its first token came after 165 to 192 ms with stock SGLang and 42 to 45 ms with Affordance.

gpt-oss-20b on SGLang did not change. We think that is because SGLang builds gpt-oss prompts with its own code instead of Jinja. On vLLM, whose tokenizer is already fast, the first token came 7% sooner on gpt-oss-20b and 3% sooner on Qwen3.6-27B.

Qwen3.6-35B-A3B on SGLang

Wait for the first token of a reply, as the conversation grows

Median of three runs, grouped by prompt size in tokens. Shorter is better.

SGLangSGLang with Affordance
under 20k
115 ms
107 ms
20k to 60k
196 ms
167 ms
60k to 120k
362 ms
291 ms
over 120k
557 ms
422 ms

The same wait, the second time the server sees each prompt

Median of three runs. Shorter is better.

SGLangSGLang with Affordance
under 20k
39 ms
30 ms
20k to 60k
58 ms
30 ms
60k to 120k
99 ms
36 ms
over 120k
167 ms
43 ms

Qwen3.6-27B on SGLang

Wait for the first token of a reply, as the conversation grows

Median of three runs. Shorter is better.

SGLangSGLang with Affordance
under 20k
377 ms
361 ms
20k to 60k
517 ms
478 ms
60k to 120k
816 ms
721 ms
over 120k
1,132 ms
968 ms

Timing the prompt step alone

We also timed the prompt step with no model running, on 200 prompts from one growing conversation. SGLang's template and tokenizer took 17.8 seconds for all 200, and Affordance took 0.47 seconds.

All runs on one machine with four RTX PRO 6000 graphics cards, SGLang v0.5.23 and vLLM v0.33.0.