Tokenization & Embeddings

Introduction

Computers do not understand language. They do not know what an "apple" is, nor do they understand the emotional weight of a Shakespearean sonnet. Computers only understand numbers—specifically, vectors of floating-point numbers.

For a neural network to process natural language (like in ChatGPT or translation apps), we must bridge this gap. We need a way to reliably convert human text into numbers that capture semantic meaning. This transformation happens in two distinct steps:

1. Tokenization: Breaking raw text into smaller, manageable chunks (tokens).
2. Embeddings: Mapping those tokens into high-dimensional mathematical space where words with similar meanings are placed physically close together.

Understanding these two concepts is the absolute prerequisite to understanding Modern Natural Language Processing (NLP) and Large Language Models (LLMs).

Core Concepts Explained Intuitively

1. Tokenization: Breaking Down the Text

If I give a computer the sentence "I love machine learning!", how should it read it?

• Character-level: ["I", " ", "l", "o", "v", "e", ...]. (Too granular, characters have no meaning on their own).
• Word-level: ["I", "love", "machine", "learning!"]. (Better, but how do we handle punctuation? What about compound words or typos?)

Modern LLMs use Subword Tokenization. This approach strikes a balance. Common words remain whole, while rare or complex words are split into smaller pieces. For example, "unbelievable" might be split into ["un", "believ", "able"]. This allows the model to handle words it has never seen before by understanding their roots and prefixes.

The most common subword algorithm is Byte-Pair Encoding (BPE) (used by OpenAI's GPT models). BPE starts by treating every character as a token. It then iteratively looks for the most frequently occurring adjacent pair of tokens and merges them into a new token. It repeats this process until it reaches a pre-defined vocabulary size (e.g., 50,000 tokens).

2. Embeddings: Mapping Meaning to Numbers

Once we have our tokens, we assign each one a unique integer ID (e.g., "apple" = 405). But to a neural network, the number 405 means nothing. Is "apple" (405) mathematically similar to "orange" (809)? No.

We need Word Embeddings. An embedding is a dense vector (a list of numbers) that represents the semantic meaning of a word. Imagine a 3D coordinate system where the X-axis is "Fruit-ness", the Y-axis is "Color", and the Z-axis is "Size".

• Apple might be mapped to [0.9, 0.8, 0.4]
• Orange might be mapped to [0.9, 0.9, 0.4]
• Car might be mapped to [0.1, 0.2, 0.9]

By plotting these coordinates, we see that Apple and Orange exist very close to each other in this "Vector Space", while Car is far away. Modern embeddings (like Word2Vec, or Transformer embeddings) don't use 3 dimensions; they use 768 or even 4096 dimensions to capture incredible nuances of language (gender, tense, royalty, geography).

3. Cosine Similarity

How do we mathematically measure how close two words are in this vector space? We use Cosine Similarity. Instead of measuring the straight-line distance between two points, we measure the angle between their vectors originating from zero.

• An angle of 0° (Cosine = 1) means identical meaning.
• An angle of 90° (Cosine = 0) means no relation.
• An angle of 180° (Cosine = -1) means opposite meaning.

Code Examples

1. Beginner Code Example: Tokenization with OpenAI's tiktoken

tiktoken is the blazing-fast BPE tokenizer used by OpenAI for models like GPT-4.

import tiktoken

# Load the encoding used by GPT-4
enc = tiktoken.encoding_for_model("gpt-4")

text = "Tokenization is fascinating! Unbelievably cool."

# Encode text into integer IDs
tokens = enc.encode(text)
print(f"Token IDs: {tokens}")
print(f"Total tokens: {len(tokens)}")

# Decode back to text to see how it was split
print("\nLet's see how the text was broken down:")
for token_id in tokens:
    # decode_single_token_bytes returns bytes, so we decode to utf-8
    chunk = enc.decode_single_token_bytes(token_id).decode('utf-8')
    print(f"[{token_id}] -> '{chunk}'")

2. Intermediate Code Example: Visualizing Word Embeddings

We will use a pre-trained embedding model via gensim to explore semantic relationships in vector space.

import gensim.downloader as api
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Load a small pre-trained Word2Vec model (GloVe trained on Twitter)
# (Note: Downloading this might take a minute the first time)
print("Loading model...")
model = api.load("glove-twitter-25") 

def check_similarity(word1, word2):
    # Retrieve the 25-dimensional vectors for both words
    vec1 = model[word1].reshape(1, -1)
    vec2 = model[word2].reshape(1, -1)
    
    # Calculate cosine similarity
    sim = cosine_similarity(vec1, vec2)[0][0]
    print(f"Similarity between '{word1}' and '{word2}': {sim:.3f}")

print("\nSemantic Relationships:")
check_similarity("king", "queen")
check_similarity("apple", "orange")
check_similarity("apple", "car")

# Vector Math! King - Man + Woman = ?
result = model.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
print(f"\nKing - Man + Woman = {result[0][0]}")

3. Advanced Code Example: Embedding Layer in PyTorch

How do neural networks handle this internally? They use an nn.Embedding layer, which is essentially a lookup table that stores vectors. These vectors are updated and learned during backpropagation.

import torch
import torch.nn as nn

# Vocabulary size: 10,000 words. Embedding dimension: 128
vocab_size = 10000
embed_dim = 128

# Create the embedding layer
embedding_layer = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embed_dim)

# Imagine a sentence of 4 words, tokenized into integer IDs
# Batch size: 1, Sequence length: 4
input_sentence = torch.tensor([[405, 809, 12, 9999]])

# Pass the tokens through the embedding layer
# The layer looks up the vector for '405', then '809', etc.
embedded_sentence = embedding_layer(input_sentence)

print(f"Input shape: {input_sentence.shape}")
print(f"Output shape: {embedded_sentence.shape}")
# Output shape is [1, 4, 128]: 1 sequence, 4 words, each represented by 128 numbers

Common Mistakes

1. Using Different Tokenizers for Training and Inference:

   • Mistake: Training a model on tokens generated by BERT's WordPiece tokenizer, and then deploying the model using OpenAI's tiktoken.
   • Fix: Token IDs are completely arbitrary mappings. The number 543 in WordPiece means something entirely different than 543 in BPE. You must use the exact same tokenizer artifact for inference that was used for training.

2. Out of Vocabulary (OOV) Errors in Word-Level Models:

   • Mistake: Using strict word-level tokenization and encountering a typo in production (e.g., "awesooome"). The model crashes because it has no ID for that word.
   • Fix: This is exactly why modern systems use subword tokenization (BPE/WordPiece). "awesooome" would just be split into ["awe", "soo", "ome"], which the model can process safely.

3. Confusing Token Length with String Length:

   • Mistake: Setting an API limit for an LLM based on character count instead of token count.
   • Fix: 1 token $\approx$ 4 characters in English, but this ratio falls apart for code, non-English languages, or complex symbols. Always count tokens using the specific library (like tiktoken) before sending requests to an LLM.

Interview Q&A

Q1: Explain the Byte-Pair Encoding (BPE) algorithm. A1: BPE is a subword tokenization method. It initializes its vocabulary with all individual characters in the training text. It then scans the text to find the most frequently adjacent pair of tokens (e.g., 'e' and 'r'). It merges them into a new token 'er' and adds it to the vocabulary. This process repeats iteratively until a specified vocabulary size is reached. This ensures common words are single tokens, while rare words are broken down into recognizable subwords.

Q2: What is the difference between One-Hot Encoding and Word Embeddings? A2: One-Hot Encoding represents words as sparse vectors where only the index of the word is 1 and all others are 0 (e.g., size equal to vocab size, 50,000 dimensions). It captures zero semantic meaning and is highly inefficient. Word Embeddings are dense, low-dimensional vectors (e.g., 256 dimensions) learned by a neural network, where the geometric relationship between vectors captures actual linguistic semantics.

Q3: How does the nn.Embedding layer work in PyTorch? Is it a mathematical function? A3: It is not a mathematical function like a Linear layer (which does matrix multiplication). It is simply a trainable lookup table. It stores a matrix of size [vocab_size, embedding_dim]. When you pass a token ID to it, it just returns the row corresponding to that ID. Because the lookup operation is tracked by autograd, those vectors are updated during backpropagation just like regular weights.

Q4: Why do we use Cosine Similarity instead of Euclidean Distance to compare embeddings? A4: Euclidean distance measures the absolute physical distance between two points. In high-dimensional text embeddings, the magnitude (length) of the vector can be skewed by word frequency (how often it appeared in training data). Cosine similarity measures the angle between vectors, disregarding magnitude. This ensures that a rare word and a common word with the exact same meaning will have a high similarity score.

Q5: What is the Word2Vec algorithm, fundamentally? A5: Word2Vec is a shallow neural network trained to predict words based on their context (or context based on a word). By forcing the network to predict that the word "bark" often appears near "dog" and "tree", the hidden layer weights naturally adjust to represent these words in a continuous vector space where semantic relationships are preserved. The weights are the embeddings.

Practice Problems

1. Problem: Use tiktoken to encode the phrase "I will be back." and "I'll be back.". Compare the resulting token lengths. Why is there a difference?
2. Problem: Using scikit-learn, calculate the cosine similarity between the vectors A = [1, 2, 3] and B = [-1, -2, -3]. What is the result and what does it tell you about the vectors' relationship?
3. Problem: Create a PyTorch nn.Embedding layer for a vocabulary of 50 words and embedding dimension of 10. Pass a tensor of shape [2, 5] (batch size 2, sequence length 5) into it. Verify the output shape matches your expectations.

Summary

• Tokenization translates raw text strings into discrete integers that neural networks can ingest.
• Modern tokenizers use Subword Algorithms (BPE) to handle rare words and typos efficiently without memory explosion.
• Embeddings map discrete token IDs into dense, high-dimensional vector spaces.
• The physical geometry of the embedding space captures human semantic meaning (e.g., King - Man + Woman = Queen).
• Cosine Similarity is the standard mathematical tool to measure how close in meaning two embedded vectors are.