English hits its target exactly — and every other language dropped along with it.

Grapheme-aware pre-tokenization (keeping matras and conjuncts glued to their base consonant) plus phrase-level merging brought all four languages within 0.26 of each other, using only 62% of the total vocabulary budget.

Fertility (X) — generated tokens ÷ raw word count, after PMI phrase-merging and BPE run to full convergence, per language.

The optimization journey

Original shared tokenizer

Vocab budget10,000 (fixed, shared)

English X1.422

Spread (max − min)5.859

Score170.7

Optimized pipeline (this report)

Vocab budget used6,224 of 10,000

English X1.200

Spread (max − min)0.257

Score3,897.1

Ranked by fertility

X = tokens / raw words

EnglishEN

English

1.200tok / word

TeluguTE

తెలుగు

1.363tok / word

HindiHI

हिन्दी

1.398tok / word

MalayalamML

മലയാളം

1.457tok / word

Vocabulary budget usage

6,224 of 10,000 tokens allocated

3,776 unused

English — 4,413 Hindi — 703 Telugu — 555 Malayalam — 553 Unused — 3,776

The full ledger

Language	Raw words	Vocab used	Tokens	Fertility (X)
English	11,890	4,413	14,268	1.200
Telugu	6,668	555	9,090	1.363
Hindi	14,420	703	20,153	1.398
Malayalam	10,018	553	14,592	1.457

English was tuned to land exactly at its 1.2 ceiling — no more vocabulary spent than needed. Telugu, Hindi, and Malayalam were run to full convergence with grapheme-aware pre-tokenization (matras and conjuncts pre-glued to their base consonant) and PMI-based phrase merging; each stopped only when no mergeable pair remained.

Spread

0.257

Score

3,897.1

Combined vocab

6,224 / 10,000

Get the tokenizer

tokenizer.json

The original tokenizer artifact this whole analysis started from — vocabulary, merges, and the four-language ratio metadata included.

1.36 MB JSON Custom multilingual BPE export

Download file ↓