Showcase / Example
One Affordance app showed five models a picture, a video or a spoken word, and all 7 answers were right
Some models can look at a picture, watch a video or listen to a sound clip. We wrote one small app that hands a model a file and asks what it is, and ran it on five models from five publishers. All 7 answers were right.
Models
5, from 5 publishersGiven
a picture, a video, a spoken wordAnswers
7 of 7 rightAffordance is an open-source program that builds the exact tokens a model reads from an app, a file written in TSX, the language of React web pages.
How a picture gets into a prompt
A model reads tokens, which are numbers that each stand for a short piece of text. A picture is not text, so the prompt holds a placeholder token at the spot where the picture belongs, and the file travels beside the prompt. A model server is the program that runs a model and answers requests. It swaps the picture in at the placeholder.
Every model family has its own placeholder and its own markers around it. These are three of them:
Qwen3.5 <|vision_start|><|image_pad|><|vision_end|> GLM-4.6V <|begin_of_image|><|image|><|end_of_image|> Gemma 4 <|image|>
Affordance has the prompt format of 740 models, and we checked each one against its publisher's official format. Of those, 237 take pictures, 170 take video and 33 take sound. For each one Affordance knows which token is the placeholder.
The app
An agent is a model that uses tools in a loop until its task is done, and a tool is a function the model can call. The app is the 23-line agent that affordance init writes, with one change: its first message is a file and a question. The file is written as a data URL, which is the file's bytes written out as text.
const first = {
kind: "input",
content: [
{ type: "image_url", image_url: { url: PICTURE } },
{ type: "text", text: task },
],
};
return <ModelContext strategy="unrolled" log={[first, ...log.records]}>
<Tool fn={answer}>{"[answer]\n"}</Tool>
</ModelContext>;A video is a video_url part and a sound clip is an input_audio part. The app does not name a model or a placeholder. ModelContext is the element that writes the prompt in the format of the model the app is run with, placeholder included. Affordance sends the file with the request.
What the models answered
We made three files: a picture filled with blue, a short video filled with green, and a recording of a voice that says the word "banana". Each model ran on SGLang, an open-source model server, on one graphics card. Affordance wrote each prompt itself and sent it as tokens, with the placeholder already in it.
| Model | Given | Asked | Answered |
|---|---|---|---|
| Qwen3.5-4B | the blue picture | What single colour fills this picture? | blue |
| Qwen3.5-4B | the green video | What single colour fills this video? | green |
| Kimi-VL-A3B-Instruct | the blue picture | What single colour fills this picture? | blue |
| LFM2.5-VL-1.6B | the blue picture | What single colour fills this picture? | blue |
| GLM-4.6V-Flash | the green video | What single colour fills this video? | green |
| Gemma 4 E2B | the blue picture | What single colour fills this picture? | blue |
| Gemma 4 E2B | the recording | What is the secret word spoken in this audio? | banana |
LFM2.5-VL has no format for tool calls, so it answered in plain text, and the agent took that text as its answer. The other four called the answer tool.
Giving a picture a size
How many tokens a picture takes depends on the model and on the picture. Qwen3.5 cuts a picture into squares of 32 by 32 pixels and each square is one token, so a 640 by 480 picture is 300 tokens and a 4K screenshot is over 8,000. A rule in the style sheet sets the most a picture may take, the way CSS sets an image's width:
img { max-tokens: 256tok; }
img[last] { max-tokens: 1024tok; }Affordance works out the largest size of the picture that the model turns into 256 tokens or fewer, resamples the file to that size, and sends the resampled file. The second rule gives the newest picture more room than the older ones, which suits an agent that takes a screenshot at every step. A sound clip is trimmed to its rule the same way.
The layout and the server have to agree on the count. For each request we compared the prompt tokens Affordance laid out with the prompt tokens the server said it processed:
| Model | What it was given | Tokens laid out | Tokens the server counted |
|---|---|---|---|
| Qwen3.5-4B | the picture as it is | 370 | 370 |
| Qwen3.5-4B | the picture, sized to 256 tokens by a rule | 562 | 562 |
| Qwen3.5-4B | the video as four frames | 598 | 598 |
| Qwen3.5-4B | the video file itself | 522 | 522 |
| Gemma 4 E2B | the picture as it is | 348 | 348 |
| Gemma 4 E2B | the recording, trimmed to 60 tokens by a rule | 155 | 155 |
A model's rule for turning a file into tokens is part of its tokenizer in Affordance. When a model has no rule for a kind of file, the file is not counted at a guess: its alt text is shown in its place, as a web page does with a picture it cannot load. The docs page on pictures, sound and video has every rule.
Try it
One script in the repository repeats this test. It starts SGLang with a model you have on disk and runs the app for each kind of file the model takes.
The same app also runs against a server that takes messages, such as vLLM or a hosted API. There the file goes inside the message and the server writes the placeholder. The harness guide in the docs says what each kind of server receives.
Each answer is from one run on SGLang 0.5.23.