The Token Audit

Sample size is the question every engagement opens with, so we measured it. Lexicons were trained on disjoint pools from 1 MB to 1.6 GB and scored against a single held-out partition, making corpus size the only variable. The answer splits cleanly: two traffic classes reach a ceiling and a modest export is enough, while source code rewards everything you can send.

Both curves above were carried past the point where they stopped moving, which is what makes the percentages meaningful: they are shares of a measured saturation point, not of the largest sample we happened to run. Prose settles early — beyond 50 MB an eightfold increase in corpus moved the ratio by 0.04%. Agentic traffic carries more structural variety and settles at 200 MB. For these two, breadth across your traffic classes is what moves the number — a representative slice, not volume for its own sake.

Source code is different, and it is worth knowing why

Code never stopped improving. We carried it to 1.6 GB — eight times the largest sample the table above needed — and every doubling of corpus was still adding compression. So we quote no ceiling for code and we do not ask for a token sample: more code keeps paying, and here is the schedule it pays on.

Read the right-hand column as the reason to send more: at 1.5 GB an extra doubling of your corpus was still worth 2.6% on every token you will ever spend. We ask code-heavy customers for 1.5 GB where the corpus supports it, and 400 MB as a working floor. A repository export reaches those numbers without a data program — source code is the one traffic class most organisations already have in volume.

The results page quotes 1.68× for developer tooling from a standard engagement. The figures above are what a code corpus returns when it is built for scale instead — which is why we ask for the larger sample.

We measure against six production tokenizers spanning four model families. The spread between them is 7% on the prose we measured and wider on agentic traffic, but either way tokenizer choice moves the result far less than domain composition does. Our published figures are quoted against o200k_base, the most efficient of the six on that prose and second on agentic traffic. Whatever you run, the comparison you receive is computed on your stack.

A model that serves one language still needs English and code, so we measured the realistic case: one vocabulary covering the language, English and source code together, scored on held-out text of each kind against the tokenizers in use.

This is public text, not your traffic, and it is a statement about token counts, not about any model trained on the vocabulary. A vocabulary for one language alone goes further — 2.15× against Gemma 3 on Ukrainian. Your number comes from your corpus.

Scope, timeline and fee are set against the size and composition of your corpus. Tell us what you run and we will come back with both.

A lexicon is built and measured on your own corpus before any figure exists — engineering against real data, held to the same standard as every number we publish. What you acquire is a quantified account of what your traffic costs you today, on your infrastructure, at your rate.

The figures are yours outright. They stand as the baseline every subsequent decision is measured against, and they are reproducible — the same corpus, measured the same way, returns the same number every time.

To be precise about what this engagement is: it is a measurement. We quantify what your traffic costs today and what it costs under a vocabulary fitted to it, and you keep the numbers. The vocabulary itself is delivered with the Lexicon Build, a separate engagement scoped once you have the numbers in hand — which is the right order, because the measurement is what tells you whether it is worth doing at all.