Tokenization & embeddings

Everything from attention onward operates on vectors of real numbers. But a prompt arrives as a string of characters. Two conversions bridge that gap, and they are the model's very first and very last contact with language. First tokenization splits the text into a sequence of known pieces — tokens — each with an integer id. Then an embedding table turns each id into a vector the network can compute with. Get these wrong and nothing downstream can recover; get them right and the model inherits a compact, meaningful input. Karpathy calls the tokenizer "a completely separate stage with its own training set" — and blames it, only half-jokingly, for most of the weird things LLMs can't do.

8.1 The two bad extremes

Why not just feed the model words? Because the vocabulary would be unbounded. Every typo, surname, product code, hashtag, and compound is a "new word," so the table could never be finished, and the long tail of rare words would each be seen too few times to learn anything. And a word the model never saw in training — its first encounter with Plasmient — would map to a single "unknown" token, erasing it.

Why not go the other way and feed raw characters? Then the vocabulary is tiny and nothing is ever unknown — but sequences explode. A paragraph becomes thousands of steps, and sinceattention costs grow with the square of the length, that is ruinous. Worse, a single character like t carries almost no meaning on its own; the model must spend depth just reassembling letters into concepts before it can reason.

Subword tokenization splits the difference: common words become a single token, while rare ones break into a few meaningful pieces. The vocabulary stays fixed and finite (typically 30k–100k entries), every possible string is still representable, and the sequence length is reasonable. The only question left iswhich pieces to use — and that is learned, not hand-designed.

8.2 Byte-Pair Encoding: a vocabulary learned by merging

Byte-Pair Encoding (BPE) chooses the pieces with a rule that is almost laughably simple. Start with every word as a list of its characters. Then scan the whole corpus, find the most frequent adjacent pair of symbols, and merge it into a single new token. Record that merge and repeat. Each pass adds one entry to the vocabulary. Sennrich et al. introduced this for machine translation in 2016; GPT-2 adapted it and it has been the default ever since.

Common sequences — t+h, then th+e — get merged early and become single tokens; rare strings stay broken into small pieces. After enough merges the frequent words are whole tokens and everything else is still representable from its parts. Type a string below (repetition helps — that's what BPE feeds on) and press merge to apply the single best merge each step. Watch the token count fall as the vocabulary grows.

Real tokenizers run BPE not on characters but on bytes — the raw UTF-8 encoding of the text. That is the trick that makes any string representable: there are only 256 possible bytes, so those are the base vocabulary, and every character on Earth (accents, Chinese, emoji) is some sequence of them. Nothing is ever truly "unknown"; the worst case is a rare emoji spelled out as four separate byte tokens. This is why the vocabulary is finite yet complete.

8.3 Subwords are the middle ground

The payoff of BPE is best seen as a comparison. Take any text and split it three ways: intocharacters (tiny vocabulary, huge sequence), into whole words (compact sequence, unbounded vocabulary, breaks on anything rare), and into subwords the way BPE does. Type below and watch: familiar words collapse to one subword token, while a long or unusual word cleanly falls back to a few meaningful parts — un+believ+ably — never to a wall of single letters, and never to a single unknown.

8.4 From id to vector: the embedding table

Tokenization gives every token an integer id, but an id like 4021 carries no structure —4022 is not "more" than it, and nothing about the number says what the token means. So the first thing the network does is a lookup: an embedding matrixE with one learned row per vocabulary entry. Token id t selects rowt — a vector of a few thousand numbers that is the token, as far as the model is concerned.

Written as onehot(t)·E it looks like a matrix multiply, but it isn't really: a one-hot vector is all zeros except a single 1, so the product just copies out one row. No multiplies, no adds — agather. Below, a sequence of tokens reads its rows out of the table. Notice that a repeated token (the appears twice) reads the same row — the work is proportional to the distinct ids touched, not the sequence length. Press play and watch the read head move.

8.5 Meaning becomes geometry

Those embedding rows start as random noise and are learned by backproplike any other weights. What emerges is the quietly astonishing part. Tokens that appear in similar contexts drift to similar vectors — the linguist J.R. Firth's old slogan, "you shall know a word by the company it keeps," made literal. This is the distributional hypothesis, and it turns meaning intogeometry: nearness is similarity, and — the result that stunned everyone in 2013 —directions carry meaning too.

Mikolov's word2vec showed that the vector from man to woman is nearly the same arrow as the one from king to queen. So you can do arithmetic on meaning:king − man + woman ≈ queen. The four words form a parallelogram. The same holds for country→capital, verb tenses, and comparatives. Below, pick a starting word and watch the model add the analogy vector (woman − man) to land on its partner. Drag the whole space or switch the relationship.

None of this is programmed in. The parallelogram falls out of predicting which words share contexts, learned by gradient descent over billions of sentences. By the time a token vector leaves the embedding table it already encodes a great deal about the word — and that vector is exactly what flows into the stack of Transformer blocks we assemble in the next chapter.