Showcase / Benchmark
We found LiquidAI's d1 reading the end of a message and the start of its question as one token, in 699 of 3,080 test messages
A language model reads and writes tokens, which are numbers that each stand for a short piece of text. A program called a tokenizer splits text into tokens, and it can join characters that sit next to each other into one token. We looked at where a customer's message meets the question after it in LiquidAI's d1, and found the message's last period joined with the newlines that follow it.
Model
LiquidAI d1Test messages
3,080Joined into one token
699d1 is a decision model: it reads a piece of text and a list of options, and picks one option. We tested 3,080 messages from Banking77, a public set of customer questions to a bank. In 699 of them, 22.7%, the end of the message and the start of the question became a single token.
The d1 prompt and its tokens
d1 calls the text it reads the state. Its prompt is the state, then three newlines and QUESTION:, then the question and its options. LiquidAI's code writes all of that as one string and tokenizes it at once. Here are the tokens for one short message, with and without a period at the end:
With the period, token 64673 holds the period and all three newlines. Tokenized on its own, the message ends with token 22, the period, and the newlines are token 7. The two versions spell the same text, and d1 was trained on the first one.
The three newlines look like a clean break between the message and the question. Whether they are depends on the message's last character. A period, ?, !, ) or } joins with them, and a letter, digit, quote or colon does not.
Six decision models compared
We checked every decision model with published weights on three public sets of short messages: Bitext customer support, Banking77 and CLINC150. Only d1-3B and d1-2B, which use LiquidAI's tokenizers, join the period with the newlines.
| Model | Joins a final period with the newlines | Messages whose tokens change: Bitext, Banking77, CLINC150 |
|---|---|---|
| LiquidAI d1-3B and d1-2B | yes | 0%, 22.7%, 0% |
| LiquidAI d1-omni-600M | no | 0, 0, 0 |
| Cloudflare Clef-flash | no | 0, 0, 0 |
| Bespoke Labs Nimble-v4-max | no | 0, 0, 0 |
| Convai laya | no | 0, 0, 0 |
Bitext and CLINC150 messages rarely end in a period, so they show 0%. When we added a period to each one, d1 joined the tokens in all of them. Clef's tokenizer keeps the period as its own token:
d1-omni-600M builds its prompt from token numbers piece by piece, with special tokens between the pieces, so its pieces never join.
LiquidAI's d1-2B code
LiquidAI's newer d1-2B comes with code that splits a prompt in two, before the question and after it, so several questions can reuse one state. Its comment explains why the split is safe:
def state_length(tokenizer, ids, suffix):
"""A prompt's tokens before its question: the prompt is the state's text then `suffix`, which opens on a token
boundary (the state ends on a blank line), so its own tokens are the prompt's last ones."""A token boundary is a place in the text where one token ends and the next begins. The code splits after QUESTION:, which is a boundary, and we found no prompt where that split went wrong. The reason the comment gives is the blank line at the end of the state, and that blank line shares a token with the period in 22.7% of Banking77 messages.
How Affordance writes d1's prompt
Affordance is an open-source program that builds the exact tokens a model reads from an app, a file written in TSX, the language of React web pages. An app shows the model a screen, and Affordance turns that screen into tokens directly, so the screen ends on its own last token. For d1 that gives token 22 and token 7 where d1 was trained on 64673.
So Affordance's d1 format writes the screen as plain text on purpose, with one call named textContent, and lets d1's tokenizer join it with the newlines the way LiquidAI's code does. A test checks Affordance's d1 prompt against LiquidAI's published prompt code, token for token.
Effect on d1's answers
We ran d1-3B and d1-2B on every message twice: tokenized LiquidAI's way, and with the message tokenized on its own. Their accuracy was the same both ways, and about 1 answer in 100 changed. Adding a period to the messages and tokenizing LiquidAI's way changed about 1 answer in 200.
| Model and message set | Messages whose tokens differ | Answers that changed |
|---|---|---|
| d1-3B, Banking77 | 699 | 8 (1.1%) |
| d1-3B, CLINC150 with periods added | 4,478 | 25 (0.6%) |
| d1-2B, Banking77 | 699 | 7 (1.0%) |
| d1-2B, CLINC150 with periods added | 4,478 | 46 (1.0%) |
We ran each model once through its publisher's code, on text messages with one question each.