Tokenizer Fairness in 2026

Following Petrov et al. (2023), the premium of a language is how many tokens its text needs relative to English for the same content (FLORES-200, parallel sentences). Premium 1.0 = as cheap as English; 15 = fifteen times more expensive. Lower is fairer.

The 2026 leaderboard

Latest tokenizer from each lab, ranked by median premium across languages (the typical language's cost). mean and the tail (p90, worst 10%) show how the low-resource languages fare. GPT-4's 2023 tokenizer is shown as the baseline.

Every language, every tokenizer

Premium for all FLORES-200 languages. Click a column header to sort; type to filter. Cell colour = premium.

What the numbers say

Method & reproduction

Every number here is reproducible. premium(L) = Σ tokens(L devtest) / Σ tokens(English devtest), no special tokens, over the FLORES-200 devtest split.

Data: FLORES-200 (200 languages, parallel). Get it via the tarball at dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz or a HuggingFace token.
Tokenizers: OpenAI via tiktoken; the rest via each model's public tokenizer.json / SentencePiece / tiktoken.model. Repos are listed in the code.
Caveats: Doubao is a closed product — its column uses ByteDance's open Seed-OSS tokenizer as a proxy. Gated labs (Meta, Google) use ungated mirrors of the identical tokenizer. Kimi's regex pattern is approximated. Latest OpenAI public tokenizer is o200k.
Code: see the repository — reproduce_tokenization_fairness.py, loaders_full.py, compute_all.py.

Reproduction & extension of Petrov, La Malfa, Torr, Bibi — “Language Model Tokenizers Introduce Unfairness Between Languages” (NeurIPS 2023). Generated 2026-07-08. Data: FLORES-200. Not affiliated with the paper's authors or any model provider.