edulab demo lab

Interactive demo · 4 min

Byte Pair Encoding

Visualization of the tokenization algorithm used in GPT and other language models

How it works

  1. Start with individual characters as tokens
  2. Find the most frequent pair of adjacent tokens
  3. Merge that pair into a new token
  4. Repeat until reaching the target vocabulary size

Change text and click outside or press Reset

Fewer More
Current Step
0
Vocabulary Size
0 / 30
Token Count
0

Current Tokens

Vocabulary (0 tokens)

Alphabet Single character tokens

BPE Merged tokens (0)

Click "Step" or "Auto" to start creating tokens
Alphabet tokens
BPE tokens
Newly created

Technical Note

This visualization omits the end-of-word marker (typically denoted as </w>) for clarity. In real BPE, each word's final character is marked with a special symbol, allowing the algorithm to distinguish tokens like "er" at the end of a word ("lower") from "er" in the middle ("merry").

This ensures BPE knows not to merge tokens across word boundaries. In this app, we use a | separator between words that serves a similar role but is hidden from display.