Tokenization and Embeddings

Tokenization is the procedure that converts raw input (text, as a sequence of bytes or characters) into a sequence of discrete symbols called tokens drawn from a fixed vocabulary. An embedding is a learned dense vector assigned to each token (or, in later models, to a whole passage), which is the numerical form on which a neural network operates. Together they define the interface between human-readable data and the Transformer layers of a language model.

A neural language model (see [TECH-LLM-001]) cannot consume a string directly. It needs a finite vocabulary of $|V|$ symbols, each mapped to a vector, and it predicts a distribution over that vocabulary at each step. The choice of vocabulary is therefore a modelling decision with consequences for several things:

• Sequence length and compute. A tokenizer that represents a text in fewer tokens makes attention and generation cheaper, because cost grows with the number of tokens (see [TECH-ARCH-001], [TECH-INFER-001]).
• Coverage. Word-level vocabularies cannot represent unseen words; characters or bytes can represent anything but yield long sequences.
• Fairness and cost. The same content can require very different token counts in different languages, which affects price, latency and usable context (Petrov et al., 2023; Ahia et al., 2023).
• Reliability. Tokens present in a vocabulary but rare or absent in training data can cause erratic behaviour (Land & Bartolo, 2024).

Embeddings matter because the geometry of the vector space determines what the model can easily generalise. The same idea extends beyond input: embedding models that map passages or images to vectors are the basis of semantic search (see [TECH-RAG-001]) and of multimodal alignment (see [TECH-MM-001]).

Before neural models, text was represented as sparse count vectors over a word vocabulary. Dense, learned word vectors became practical with the word2vec models of Mikolov et al. (2013a, 2013b), which learned vectors from large corpora fast enough to use 1.6 billion words in under a day and evaluated them on syntactic and semantic similarity. GloVe (Pennington, Socher & Manning, 2014) fitted vectors to global co-occurrence statistics. fastText (Bojanowski et al., 2017) composed word vectors from character n-grams so that rare and unseen words obtain representations.

The open-vocabulary problem in neural machine translation led Sennrich, Haddow and Birch (2016) to adopt a segmentation based on the byte pair encoding (BPE) compression algorithm, encoding rare words as sequences of subword units; they reported gains of 1.1 and 1.3 BLEU over a back-off dictionary on WMT 15 English–German and English–Russian. Wu et al. (2016) described a related wordpiece approach for Google's translation system. Kudo (2018) proposed a unigram-language-model segmentation with subword regularisation, and Kudo and Richardson (2018) released SentencePiece, which trains directly on raw text without requiring a pre-tokenised word sequence. BERT (Devlin et al., 2019) used a subword vocabulary together with contextual representations, replacing static word vectors with ones that depend on the surrounding sentence.

More recently, token-free approaches processing bytes directly have been studied with standard Transformers (ByT5; Xue et al., 2022) and, for the Byte Latent Transformer, in a scaling study up to 8B parameters (Pagnoni et al., 2024). Subword tokenizers remain the documented norm in the models reviewed for this article; this article does not survey whether later frontier models adopted byte-level or patch-based designs.

#4.1 Byte-pair encoding

BPE learns a vocabulary from a training corpus. In its subword form (Sennrich et al., 2016):

1. Start with a vocabulary of base symbols (characters, or the 256 possible byte values in byte-level variants).
2. Count all adjacent symbol pairs in the corpus.
3. Merge the most frequent pair into a new symbol and add it to the vocabulary; record the merge rule.
4. Repeat until the vocabulary reaches a target size.

At encoding time the learned merges are applied to new text in the order in which they were learned. Frequent words become single tokens; rare words decompose into several pieces. Byte-level BPE, which builds the vocabulary on bytes, was described for machine translation by Wang, Cho and Gu (2019); because it starts from bytes it produces no out-of-vocabulary tokens, so any Unicode string is representable. Sennrich et al. (2016) describe the subword BPE method itself, not the byte-level variant.

#4.2 Unigram and WordPiece variants

The unigram model (Kudo, 2018) starts from a large candidate vocabulary and prunes it to maximise the likelihood of the training data under a unigram language model; it can produce several segmentations of the same string with different probabilities, which subword regularisation uses as training noise. WordPiece (Wu et al., 2016) is likewise a data-driven subword scheme. The families differ in how the vocabulary is built and in how ties and ambiguities are resolved; all of them produce a deterministic or sampled mapping from a string to a sequence of vocabulary indices.

#4.3 The embedding layer

Given token indices $t_1,\dots,t_n$ with $t_i \in \{1,\dots,|V|\}$ and an embedding matrix $E \in \mathbb{R}^{|V|\times d}$, the input to the Transformer is the matrix whose $i$-th row is $x_i = E_{t_i}$, with positional information added or applied inside attention (Vaswani et al., 2017; see [TECH-ARCH-001]). The matrix $E$ is a trained parameter, so $|V|\cdot d$ parameters are spent on it; the output projection back to $|V|$ logits has the same shape. A language model is trained to minimise the cross-entropy

over tokens, so its units of prediction are the tokenizer's tokens.

#4.4 Why tokenizers complicate comparison

Because $\mathcal{L}$ is measured per token, models with different tokenizers have different units. If a tokenizer yields on average $\tau$ tokens per byte of text and the loss in nats per token is $\ell$, the loss per byte in bits is $\ell\,\tau / \ln 2$. Comparisons across tokenizers should therefore use a per-byte or per-character measure, not per-token perplexity. This is a direct consequence of the definitions and is not a claim about any specific model.

#4.5 Static versus contextual embeddings

word2vec, GloVe and fastText assign one vector per word type, so a polysemous word has one vector. A Transformer layer stack outputs contextual vectors in which the same token receives different representations in different sentences (Devlin et al., 2019). Sentence-level embedding models, such as Sentence-BERT (Reimers & Gurevych, 2019), pool or fine-tune such representations so that semantically similar sentences have high cosine similarity; embedding-based retrieval uses the same idea of comparing learned vectors, though dense passage retrievers such as that of Karpukhin et al. (2020) train separate query and passage encoders with a similarity score over their outputs.

#4.6 Vocabulary size trade-off

A larger vocabulary shortens sequences but enlarges the embedding and output matrices and makes each rare token less frequently trained. As one documented data point, the Llama 3 report describes a 128K-token vocabulary (100K tokens from the tiktoken tokenizer plus 28K additional tokens for non-English languages) and states that, versus the Llama 2 tokenizer, compression on a sample of English data improved from 3.17 to 3.94 characters per token (Grattafiori et al., 2024). This is a single developer-reported figure on one sample; it is not a general law.

The Byte Latent Transformer (Pagnoni et al., 2024) groups bytes into patches whose lengths depend on the entropy of the next byte; the authors report the first FLOP-controlled scaling study of byte-level models up to 8B parameters and 4T training bytes and better scaling than token-based models at fixed inference cost. This is single-group evidence (Preliminary).

• Text LLMs. The widely used autoregressive language models reviewed for this article use a tokenizer, with the vocabulary and algorithm usually published for open-weight models. For closed models the vocabulary may be exposed through the API or a library, but whether the training-time tokenizer matches a published one is a developer matter; this article does not infer it.
• Billing. Commercial APIs commonly price by tokens. Ahia et al. (2023) analysed OpenAI's API on multilingual benchmarks in 22 languages and report that speakers of many of the languages are overcharged while obtaining poorer results, because of differences in token counts.
• Retrieval. Dense passage encoders map queries and documents to vectors for nearest-neighbour search (Reimers & Gurevych, 2019; Karpukhin et al., 2020).
• Cross-modal embeddings. CLIP-style models align image and text embeddings in a shared space (see [TECH-MM-001]).

#Advantages of subword tokenization

Fixed, moderate sequence lengths; no unseen-word failure; a mature toolchain (Kudo & Richardson, 2018).

#Limitations

• Cross-language disparity. Petrov et al. (2023) report that the same text translated into different languages can differ in tokenization length by up to 15 times in some cases, even for tokenizers built for multilingual support, with character- and byte-level models still showing over 4x differences for some language pairs. They argue this produces unequal cost, latency and available context (see [SAFE-ETHICS-001]).
• Tokenizer quality affects downstream performance. Rust et al. (2021) examine monolingual performance of multilingual models as a function of tokenizer quality.
• Under-trained ("glitch") tokens. Land and Bartolo (2024) present methods to detect vocabulary tokens that are nearly or entirely absent from training data and show such tokens are prevalent across a diverse set of models; these can induce unwanted behaviour.
• Opaque units. Models see tokens, not characters. Tasks that depend on spelling or on character-level structure can be harder; the extent for any given model must be measured, not assumed.

#Unresolved questions

1. Whether tokenizer-free models will match subword models at the largest scales; current evidence is limited to the studies cited (Preliminary).
2. How to design vocabularies that are fair across languages without large cost in compute.
3. How much of the geometric structure of embedding spaces reflects meaning rather than corpus statistics; the word-vector literature measures similarity tasks and does not settle this.

As of 2026-10-19, subword tokenization (BPE variants and SentencePiece-style unigram or BPE models) is the standard in the documented language models reviewed here (no survey of 2026 model documentation was performed), and byte-level BPE is a documented option for coverage. Dense learned embeddings, both input embeddings and sentence-level retrieval embeddings, are standard components. Claims about the tokenizer or vocabulary of a specific closed model should be taken only from developer documentation or from the model's public tokenizer, and are otherwise undisclosed. Current vocabulary sizes and compression rates of named models belong in the Living Reference (MODEL-001), not in this article.

Observed directions: dynamic or learned segmentation (Pagnoni et al., 2024), explicit tests for under-trained tokens (Land & Bartolo, 2024), and fairness-aware tokenizer design (Petrov et al., 2023). Whether byte-level or patch-based models become the default is open; no forecast is registered here.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The models that consume tokens.
  • TECH-ARCH-001 The Transformer Architecture: Where embeddings and positional information enter the network.
  • TECH-NN-001 Artificial Neural Networks: Learned representations in general.
• Related Methods & Technologies:
  • TECH-ATTN-001 Attention Mechanisms: Cost of attention grows with token count.
  • TECH-RAG-001 Retrieval-Augmented Generation: Embedding-based retrieval.
  • TECH-MM-001 Multimodal Foundation Models: Image and audio tokens and shared embedding spaces.
  • TECH-INFER-001 Inference, Quantization and Efficiency: Token count drives serving cost.
• Advanced Implications & Future Trajectories:
  • SAFE-ETHICS-001 AI Ethics, Bias and Fairness: Language-community disparities.