Tokenizer Visualizer

About Tokenizer Visualizer

How do LLMs actually read your words? Paste any Chinese, English or code and this tool loads OpenAI's public tokenizers (o200k_base / cl100k_base) to split it into color-highlighted tokens — one chip per token, hover for its id. Common Chinese words are often a single token; rare ones shatter into pieces. You also get exact and unique token counts, the chars/token ratio, a most-frequent-tokens board, and a one-click comparison between the two encodings (GPT-4o-era vs GPT-3.5/4). Models without public vocabularies (Claude, Gemini) only get clearly-labelled estimates. Everything runs locally in your browser.

FAQ

Why is "你好" one token while a rare character shatters into pieces?

Byte-pair encoding merges text by co-occurrence frequency in the training corpus: frequent pairs earn a vocabulary slot (many common Chinese words and English subwords did), while rare characters must be spelled from byte-level fragments — 2–4 tokens for one character is common. That's exactly why the same sentence can cost different token counts on different models, and part of your API bill variance.

What's the difference between o200k_base and cl100k_base?

cl100k_base is the GPT-3.5/GPT-4 era encoding (~100K vocab) and is relatively inefficient for Chinese; o200k_base ships from GPT-4o onward (~200K vocab) with much better non-English coverage — the same Chinese text usually costs fewer tokens. The tool's compare button loads both and shows the delta.

Can I see Claude's or Gemini's segmentation?

No — Anthropic and Google don't publish their vocabularies; any tool claiming "exact Claude tokenization" is making it up. For those models this tool only offers clearly-labelled estimates from published char/token ratios. Only OpenAI publishes encodings (o200k/cl100k), which is why exact visualization covers exactly those two.