Measured, per sector
Each lexicon is trained on one sector's traffic and measured on held-out text from that same sector, split by content hash, with the round trip verified lossless. Eleven sectors, every one of them trained on a matched corpus of 50 MB or more.
Against tiktoken's o200k_base, the tokenizer behind GPT-4o. Higher is fewer tokens for the same bytes.
Two layers, kept deliberately separate: token counts are measured; dollars are that measurement times a published rate you can substitute for your own.
| Sector | Tokens / TB today | Reduction | Tokens saved | Value / TB |
|---|
Every row reconciles from the first column: value = (tokens today − tokens today ÷ reduction) × $3 per million. The dollar figure deliberately does not track the reduction, because sectors do not start from the same place — o200k reads physics prose at 5.20 bytes per token and financial text at 3.80, so a terabyte of finance arrives as 263 billion tokens against physics’s 192 billion. Denser traffic has more to remove at any ratio. Scale tiers multiply directly: 1 TB mid-size deployment, 10 TB large enterprise, 100 TB platform, 1,000 TB hyperscale. Which layer applies depends on how you run: if you consume a third-party API the provider counts with the provider’s own tokenizer, so that invoice is unchanged — the dollar column above values the tokens removed at a market rate, it does not predict a smaller bill. If you run your own inference the saving is prefill compute, and the table below puts it in that unit.
A context window is counted in tokens, not in meaning. On a model adapted to the lexicon, fewer tokens for the same material is a larger effective window at no extra cost per request.
Shown at the measured 2.916× agentic ratio. Traffic that mixes classes in one stream measures 1.97×; tuned to a single customer's own traffic it reaches 3.14×.
Six production tokenizers, four families, one held-out corpus. The comparison does not depend on which vendor you are on.
Published figures on this site are quoted against GPT-4o's o200k_base, which is the most token-efficient of the baselines tested — so it is the hardest comparison available. Against every other tokenizer measured, the same lexicon does better.
| Tokenizer | Family | Tokens for the same text | We need |
|---|---|---|---|
| p50k_base | OpenAI | 1,158,821 | 2.03× fewer |
| Qwen3 | Open weights | 1,101,025 | 1.93× fewer |
| cl100k_base | OpenAI | 1,094,765 | 1.92× fewer |
| SmolLM3 | Open weights | 1,093,745 | 1.92× fewer |
| OLMoE | Open weights | 1,087,037 | 1.91× fewer |
| o200k_base | OpenAI — GPT-4o | 1,082,031 | 1.90× fewer |
The spread across four tokenizer families is 1.90× to 2.03× — a range of 7%. Tokenizer choice moves the result far less than domain does, which is why the engagement measures your corpus rather than your vendor.
For operators running their own inference. The tokens removed are measured; the two rates are reference values you substitute for your own.
| Sector | Tokens removed / TB | GPU-hours saved | At $2.00 / GPU-hour |
|---|
Read this table and the one above as two denominations of
one measurement, not two findings. Both are linear in the tokens removed, so the ratio between them
is a constant — a token valued at $3.00 per million is worth about
54× what it costs to prefill at these reference rates. That gap is
the whole reason to be explicit about which one applies to you.
Two honest edges. This is a floor. The model is linear in positions, while attention cost
grows faster than linearly in sequence length — so at long context, removing a given fraction
of positions removes more than that fraction of the attention term. We publish the linear figure
because it is the one we can reconcile. And throughput here is a reference, not our
measurement. Prefill throughput on a transplanted model has not been measured; substitute the
number your own stack produces. For many operators the binding constraint is not the dollar line at
all but capacity — requests per GPU, or whether a context fits.
Measured the same way: held-out data, and the same text or images for every comparison.
One vocabulary covering each language, English and code, against Sarvam-105B, Gemma 3 and EuroLLM-22B on public text, with every held-out document byte-exact.
With no measurable change in answers on two open vision models, and 3.07× on rendered text pages with a pixel-identical round trip.