affordance

Showcase  /  Example

One Affordance app showed five models a picture, a video or a spoken word, and all 7 answers were right

Some models can look at a picture, watch a video or listen to a sound clip. We wrote one small app that hands a model a file and asks what it is, and ran it on five models from five publishers. All 7 answers were right.

Models

5, from 5 publishers

Given

a picture, a video, a spoken word

Answers

7 of 7 right

Affordance is an open-source program that builds the exact tokens a model reads from an app, a file written in TSX, the language of React web pages.

How a picture gets into a prompt

A model reads tokens, which are numbers that each stand for a short piece of text. A picture is not text, so the prompt holds a placeholder token at the spot where the picture belongs, and the file travels beside the prompt. A model server is the program that runs a model and answers requests. It swaps the picture in at the placeholder.

Every model family has its own placeholder and its own markers around it. These are three of them:

Qwen3.5
<|vision_start|><|image_pad|><|vision_end|>

GLM-4.6V
<|begin_of_image|><|image|><|end_of_image|>

Gemma 4
<|image|>

Affordance has the prompt format of 740 models, and we checked each one against its publisher's official format. Of those, 237 take pictures, 170 take video and 33 take sound. For each one Affordance knows which token is the placeholder.

The app

An agent is a model that uses tools in a loop until its task is done, and a tool is a function the model can call. The app is the 23-line agent that affordance init writes, with one change: its first message is a file and a question. The file is written as a data URL, which is the file's bytes written out as text.

const first = {
  kind: "input",
  content: [
    { type: "image_url", image_url: { url: PICTURE } },
    { type: "text", text: task },
  ],
};
return <ModelContext strategy="unrolled" log={[first, ...log.records]}>
  <Tool fn={answer}>{"[answer]\n"}</Tool>
</ModelContext>;

A video is a video_url part and a sound clip is an input_audio part. The app does not name a model or a placeholder. ModelContext is the element that writes the prompt in the format of the model the app is run with, placeholder included. Affordance sends the file with the request.

What the models answered

We made three files: a picture filled with blue, a short video filled with green, and a recording of a voice that says the word "banana". Each model ran on SGLang, an open-source model server, on one graphics card. Affordance wrote each prompt itself and sent it as tokens, with the placeholder already in it.

ModelGivenAskedAnswered
Qwen3.5-4Bthe blue pictureWhat single colour fills this picture?blue
Qwen3.5-4Bthe green videoWhat single colour fills this video?green
Kimi-VL-A3B-Instructthe blue pictureWhat single colour fills this picture?blue
LFM2.5-VL-1.6Bthe blue pictureWhat single colour fills this picture?blue
GLM-4.6V-Flashthe green videoWhat single colour fills this video?green
Gemma 4 E2Bthe blue pictureWhat single colour fills this picture?blue
Gemma 4 E2Bthe recordingWhat is the secret word spoken in this audio?banana

LFM2.5-VL has no format for tool calls, so it answered in plain text, and the agent took that text as its answer. The other four called the answer tool.

Giving a picture a size

How many tokens a picture takes depends on the model and on the picture. Qwen3.5 cuts a picture into squares of 32 by 32 pixels and each square is one token, so a 640 by 480 picture is 300 tokens and a 4K screenshot is over 8,000. A rule in the style sheet sets the most a picture may take, the way CSS sets an image's width:

img       { max-tokens: 256tok; }
img[last] { max-tokens: 1024tok; }

Affordance works out the largest size of the picture that the model turns into 256 tokens or fewer, resamples the file to that size, and sends the resampled file. The second rule gives the newest picture more room than the older ones, which suits an agent that takes a screenshot at every step. A sound clip is trimmed to its rule the same way.

The layout and the server have to agree on the count. For each request we compared the prompt tokens Affordance laid out with the prompt tokens the server said it processed:

ModelWhat it was givenTokens laid outTokens the server counted
Qwen3.5-4Bthe picture as it is370370
Qwen3.5-4Bthe picture, sized to 256 tokens by a rule562562
Qwen3.5-4Bthe video as four frames598598
Qwen3.5-4Bthe video file itself522522
Gemma 4 E2Bthe picture as it is348348
Gemma 4 E2Bthe recording, trimmed to 60 tokens by a rule155155

A model's rule for turning a file into tokens is part of its tokenizer in Affordance. When a model has no rule for a kind of file, the file is not counted at a guess: its alt text is shown in its place, as a web page does with a picture it cannot load. The docs page on pictures, sound and video has every rule.

Try it

One script in the repository repeats this test. It starts SGLang with a model you have on disk and runs the app for each kind of file the model takes.

$ MODEL=/path/to/model TEMPLATE=Qwen/Qwen3.5-4B ./tools/check-media.sh

The same app also runs against a server that takes messages, such as vLLM or a hosted API. There the file goes inside the message and the server writes the placeholder. The harness guide in the docs says what each kind of server receives.

Each answer is from one run on SGLang 0.5.23.