--- id: TECH-TOK-001 title: "Tokenization and Embeddings" alternative_names: - "Subword Tokenization" - "Byte-Pair Encoding and Word Embeddings" - "Input Representation in Language Models" category: "Technical → Language Models" volume: III chapter: 19 entry_type: "Technical Article" level: 3 status: "Draft" last_reviewed: 2026-10-15 version: "0.1.0" prerequisites: - "TECH-LLM-001" - "TECH-ARCH-001" related_entries: - "TECH-NN-001" - "TECH-ATTN-001" - "TECH-RAG-001" - "TECH-MM-001" - "TECH-INFER-001" - "SAFE-ETHICS-001" --- # Tokenization and Embeddings ## 1. Concise Definition **Tokenization** is the procedure that converts raw input (text, as a sequence of bytes or characters) into a sequence of discrete symbols called **tokens** drawn from a fixed vocabulary. An **embedding** is a learned dense vector assigned to each token (or, in later models, to a whole passage), which is the numerical form on which a neural network operates. Together they define the interface between human-readable data and the Transformer layers of a language model. ## 2. Overview & Significance A neural language model (see [TECH-LLM-001]) cannot consume a string directly. It needs a finite vocabulary of $|V|$ symbols, each mapped to a vector, and it predicts a distribution over that vocabulary at each step. The choice of vocabulary is therefore a modelling decision with consequences for several things: - **Sequence length and compute.** A tokenizer that represents a text in fewer tokens makes attention and generation cheaper, because cost grows with the number of tokens (see [TECH-ARCH-001], [TECH-INFER-001]). - **Coverage.** Word-level vocabularies cannot represent unseen words; characters or bytes can represent anything but yield long sequences. - **Fairness and cost.** The same content can require very different token counts in different languages, which affects price, latency and usable context (Petrov et al., 2023; Ahia et al., 2023). - **Reliability.** Tokens present in a vocabulary but rare or absent in training data can cause erratic behaviour (Land & Bartolo, 2024). Embeddings matter because the geometry of the vector space determines what the model can easily generalise. The same idea extends beyond input: embedding models that map passages or images to vectors are the basis of semantic search (see [TECH-RAG-001]) and of multimodal alignment (see [TECH-MM-001]). ## 3. Historical Development Before neural models, text was represented as sparse count vectors over a word vocabulary. Dense, learned word vectors became practical with the word2vec models of Mikolov et al. (2013a, 2013b), which learned vectors from large corpora fast enough to use 1.6 billion words in under a day and evaluated them on syntactic and semantic similarity. GloVe (Pennington, Socher & Manning, 2014) fitted vectors to global co-occurrence statistics. fastText (Bojanowski et al., 2017) composed word vectors from character n-grams so that rare and unseen words obtain representations. The open-vocabulary problem in neural machine translation led Sennrich, Haddow and Birch (2016) to adopt a segmentation based on the byte pair encoding (BPE) compression algorithm, encoding rare words as sequences of subword units; they reported gains of 1.1 and 1.3 BLEU over a back-off dictionary on WMT 15 English–German and English–Russian. Wu et al. (2016) described a related wordpiece approach for Google's translation system. Kudo (2018) proposed a unigram-language-model segmentation with subword regularisation, and Kudo and Richardson (2018) released SentencePiece, which trains directly on raw text without requiring a pre-tokenised word sequence. BERT (Devlin et al., 2019) used a subword vocabulary together with contextual representations, replacing static word vectors with ones that depend on the surrounding sentence. More recently, token-free approaches processing bytes directly have been scaled to billions of parameters (Xue et al., 2022; Pagnoni et al., 2024), without displacing subword tokenizers in widely documented models as of 2026-10-15. ## 4. Technical / Conceptual Explanation ### 4.1 Byte-pair encoding BPE learns a vocabulary from a training corpus. In its subword form (Sennrich et al., 2016): 1. Start with a vocabulary of base symbols (characters, or the 256 possible byte values in byte-level variants). 2. Count all adjacent symbol pairs in the corpus. 3. Merge the most frequent pair into a new symbol and add it to the vocabulary; record the merge rule. 4. Repeat until the vocabulary reaches a target size. At encoding time the learned merges are applied to new text in the order in which they were learned. Frequent words become single tokens; rare words decompose into several pieces. Byte-level BPE starts from bytes so that any Unicode string, including text never seen in training, is representable. ### 4.2 Unigram and WordPiece variants The unigram model (Kudo, 2018) starts from a large candidate vocabulary and prunes it to maximise the likelihood of the training data under a unigram language model; it can produce several segmentations of the same string with different probabilities, which subword regularisation uses as training noise. WordPiece (Wu et al., 2016) is likewise a data-driven subword scheme. The families differ in how the vocabulary is built and in how ties and ambiguities are resolved; all of them produce a deterministic or sampled mapping from a string to a sequence of vocabulary indices. ### 4.3 The embedding layer Given token indices $t_1,\dots,t_n$ with $t_i \in \{1,\dots,|V|\}$ and an embedding matrix $E \in \mathbb{R}^{|V|\times d}$, the input to the Transformer is the matrix whose $i$-th row is $x_i = E_{t_i}$, with positional information added or applied inside attention (Vaswani et al., 2017; see [TECH-ARCH-001]). The matrix $E$ is a trained parameter, so $|V|\cdot d$ parameters are spent on it; the output projection back to $|V|$ logits has the same shape and some models share the two matrices. A language model is trained to minimise the cross-entropy $$ \mathcal{L} = -\frac{1}{n}\sum_{i=1}^{n} \log p_\theta(t_i \mid t_{