Measured, per sector

Results

Each lexicon is trained on one sector's traffic and measured on held-out text from that same sector, split by content hash, with the round trip verified lossless. Eleven sectors, every one of them trained on a matched corpus of 50 MB or more.

Token reduction on held-out traffic

Against tiktoken's o200k_base, the tokenizer behind GPT-4o. Higher is fewer tokens for the same bytes.

1.0×1.5×2.0×2.5×3.0×

Agentic & reasoning traffic Technical & document domains

What the reduction is worth

Two layers, kept deliberately separate: token counts are measured; dollars are that measurement times a published rate you can substitute for your own.

Per terabyte of traffic, at $3.00 per million tokens.
Sector	Tokens / TB today	Reduction	Tokens saved	Value / TB

Every row reconciles from the first column: value = (tokens today − tokens today ÷ reduction) × $3 per million. The dollar figure deliberately does not track the reduction, because sectors do not start from the same place — o200k reads physics prose at 5.20 bytes per token and financial text at 3.80, so a terabyte of finance arrives as 263 billion tokens against physics’s 192 billion. Denser traffic has more to remove at any ratio. Scale tiers multiply directly: 1 TB mid-size deployment, 10 TB large enterprise, 100 TB platform, 1,000 TB hyperscale. Both layers hold whether you are paying an API invoice or your own prefill compute.

The same window, carrying more

A context window is counted in tokens, not in meaning. Fewer tokens for the same material is a larger effective window at no extra cost.

Shown at the measured 2.916× agentic ratio. Traffic that mixes classes in one stream measures 1.97×; tuned to a single customer's own traffic it reaches 3.14×.

Measured against whatever you already run

Six production tokenizers, four families, one held-out corpus. The comparison does not depend on which vendor you are on.

Published figures on this site are quoted against GPT-4o's o200k_base, which is the most token-efficient of the baselines tested — so it is the hardest comparison available. Against every other tokenizer measured, the same lexicon does better.

One sector's held-out corpus, encoded by each tokenizer. Higher means we need proportionally fewer tokens for identical text.
Tokenizer	Family	Tokens for the same text	We need
p50k_base	OpenAI	1,158,821	2.03× fewer
Qwen3	Open weights	1,101,025	1.93× fewer
cl100k_base	OpenAI	1,094,765	1.92× fewer
SmolLM3	Open weights	1,093,745	1.92× fewer
OLMoE	Open weights	1,087,037	1.91× fewer
o200k_base	OpenAI — GPT-4o	1,082,031	1.90× fewer

The spread across four tokenizer families is 1.90× to 2.03× — a range of 7%. Tokenizer choice moves the result far less than domain does, which is why the engagement measures your corpus rather than your vendor.

How to read these figures

Compression and privacyA tuned vocabulary changes what a request costs and who can read the token stream. Every claim here is sized to what has been measured.

Single-class versus mixedThe 3.14× case is tuned to one customer's own traffic, trained on 75 MB of it. Where traffic mixes classes in a single stream — prose, code and agentic together — a held-out blend of all three measures 1.97×. We quote whichever matches the traffic in question, and every figure is measured on your own corpus before it is quoted.

Your corpus sets your numberGains vary by domain, so the figure that matters is the one measured on your own traffic — and that measurement comes before any commitment.

Provenance travels with each rowTraining corpus size is published beside every sector result. Reasoning & agent traces is measured on a corpus assembled from public chain-of-thought datasets.