LLM internals: tokens, embeddings, attention

Transformer-based LLMs represent text as sequences of vectors, process them through layers of attention and feedforward operations, and produce output one token at a time. This covers the internal mechanics: how text becomes numbers, how context resolves ambiguity, and how the architecture is structured across the three main model families.

Tokenization

The first step is splitting text into tokens — not necessarily words. Modern LLMs use subword tokenization (typically Byte Pair Encoding or SentencePiece), which sits between two extremes that each have problems.

Word-level tokenization would need a vocabulary of millions — every proper noun, conjugated verb, misspelling, and compound across every language would need its own entry. “running”, “runs”, and “ran” would be three unrelated tokens with no shared structure. Character-level tokenization avoids that but produces sequences four to five times longer, which is expensive for attention (the cost scales quadratically with sequence length), and individual characters carry no inherent semantic weight.

Subword tokenization merges the most frequent adjacent character pairs iteratively until a target vocabulary size is reached. Starting from individual characters, “un” becomes a token because it’s common; then “believe”; then “unbelievable” if it appears frequently enough — or the training data might not have it often, leaving the subword split. The vocabulary reflects corpus frequency.

"unbelievable" → ["un", "believe", "able"]
"tokenization" → ["token", "ization"]
"GPT"          → ["G", "PT"]

Each token maps to an integer token ID via a fixed vocabulary lookup table. “bank” might be token ID 5234; “river” might be 8891. GPT-2 and GPT-3 use 50,257 tokens; LLaMA uses 32,000.

Practical implications for developers:

• Token count ≠ word count. A rough rule of thumb is 1 token ≈ 0.75 English words, but this varies by domain.
• Numbers fragment unpredictably. “2024” might be one token or several depending on training frequency.
• Capitalization often produces a different token ID: “Bank” and “bank” are usually distinct entries.
• Code is more fragmented than prose — variable names and operators split at unexpected boundaries.
• Context window limits are in tokens, not words. A 128k-token context holds roughly 90,000 words of English.

Token IDs are the inputs to the model. Everything downstream operates on these integers converted to vectors.

Embeddings

The embedding layer converts each token ID into a dense vector via a lookup table — mathematically a multiplication of a one-hot vector by an embedding matrix. The result is a fixed-size vector of floating-point numbers, typically 768 to 12,288 dimensions depending on the model.

Token ID 5234 ("bank") → [0.24, -0.13, 0.87, 0.45, ..., -0.32, 0.61]
                           dim1   dim2   dim3   dim4         dim767 dim768

These dimensions don’t have explicit labels — no one decided dimension 42 means “water-related”. The values are learned.

During training, the model’s task is simple to state: given a sequence of words, guess the next one. It starts with random numbers everywhere, including in the embedding table, and is mostly wrong. Each wrong guess triggers a correction: every number that contributed to the bad prediction gets nudged slightly in the direction that would have made a better one. Repeat this across hundreds of billions of words of text, and the numbers stabilize into values that make the model consistently better at the task.

What comes out of this process is that words appearing in similar contexts end up with similar vectors. “river”, “lake”, and “stream” tend to appear near similar words throughout the training text — so after enough corrections, their embeddings converge. Dimension 42 didn’t get labeled “water”; it just happens that adjusting it upward consistently helped the model predict the right next word whenever water-related tokens were involved.

“bank” is a harder case: it appears in both water contexts (“river bank”, “bank of fog”) and financial ones (“bank statement”, “bank robbery”). The model can’t store two separate embeddings for it — there’s one row per token in the table. What it settles on is a compromise that carries a signal for both meanings, which is why multiple dimension clusters end up with moderate values rather than one cluster dominating. The ambiguity gets resolved later, by attention.

This initial vector is context-independent — the same regardless of where the word appears. “bank” in “river bank” and “bank” in “bank account” start with identical embeddings. Context is resolved later by the attention mechanism.

Dimension sizes by model

Model	Embedding dimensions	Vocabulary	Parameters
BERT-base	768	30,522	110M
GPT-2 XL	1,600	50,257	1.5B
GPT-3	12,288	50,257	175B
LLaMA-2 70B	8,192	32,000	70B

Larger embedding dimensions allow more nuanced feature representation but increase memory and compute costs. The embedding matrix alone for GPT-3 is 50,257 × 12,288 ≈ 617M parameters — the remaining ~174B live in attention and feedforward layers.

Positional encoding

Attention processes all tokens simultaneously — it has no inherent sense of order. Without positional information, “the dog chased the cat” and “the cat chased the dog” would produce identical representations.

The solution is to generate a unique numeric fingerprint for each position in the input token sequence, and add it to that position’s token embedding before processing. The result is that the same word at position 3 and the same word at position 7 enter the transformer as slightly different vectors.

Doesn’t adding a fingerprint corrupt the meaning already in the vector?

It would — if the model had first learned “clean” semantic embeddings and then had position added on top. But that’s not the order of events. The model was trained with positional encoding applied from the very first iteration. Every weight matrix downstream (Q, K, V projections, feedforward layers, all of it) learned to work with position-encoded inputs from scratch. There is no clean state being overwritten; the combined signal is the only input the model has ever seen.

Why don’t the two signals destroy each other? High-dimensional vectors (768+ dimensions) have enough room that semantic content and positional content settle into different parts of the vector through training. The learned weight matrices act as projections: when attention wants to compare meaning, it projects along dimensions that carry semantic signal; when it needs position, along positional ones. Both coexist; the projections decide which to read.

The fingerprint

The fingerprint for each position is computed using sine and cosine waves at different frequencies. Some dimensions cycle fast — their values change noticeably between adjacent positions — which lets the model distinguish nearby words. Others cycle slowly, barely moving across hundreds of positions, encoding a rough sense of “early vs. late in the sequence”. Together they form a signature that is unique at every position:

Position 1:  [0.00,  1.00,  0.00,  1.00,  0.00,  1.00, ...]
Position 2:  [0.84,  0.54,  0.01,  1.00,  0.00,  1.00, ...]
Position 3:  [0.91, -0.42,  0.02,  1.00,  0.00,  1.00, ...]
Position 5:  [-0.76, -0.65,  0.04,  1.00,  0.00,  1.00, ...]

The leftmost values change fast (0.00 → 0.84 → 0.91 → -0.76); the rightmost barely move (1.00 throughout). No two rows are identical, and nearby rows are more similar to each other than distant rows — both properties the model can exploit.

The fingerprint is added to the token’s embedding before processing:

"bank" embedding:    [0.24, -0.13,  0.87,  0.45, ...]
+ position 3:        [0.91, -0.42,  0.02,  1.00, ...]
= combined:          [1.15, -0.55,  0.89,  1.45, ...]

RoPE (Rotary Position Embedding) avoids the addition step entirely. Instead of modifying the embedding, it rotates the query and key vectors just before the dot product in attention — injecting positional information at the exact moment it’s needed, without touching the embedding at all. Most current models (LLaMA, Mistral, Gemma) use RoPE for this reason, and it handles context lengths beyond training more gracefully than sinusoidal encoding.

Attention

The embedding for “bank” carries both the financial and geographic meanings simultaneously. To use the word correctly, the model needs to look at surrounding words — “river” and “steep” steer it toward geography; “loan” and “approved” steer it toward finance. Attention is the mechanism that performs this lookup: for each token in the sequence, it computes how relevant every other token is, then produces a new vector that blends in the most relevant neighbors’ information.

Q, K, V

For each token, the model computes three vectors by multiplying the token’s embedding by three separate learned matrices:

• Q (query) — a projection representing what this token is looking for in its context
• K (key) — a projection representing what this token offers to others for comparison
• V (value) — the information this token contributes if it’s found relevant

Why three separate matrices rather than comparing embeddings directly? The embedding is a general-purpose vector — it also feeds into the feedforward layer, flows through residual connections, and serves as the surface that other tokens attend to. Q and K project it into a space specifically shaped for relevance comparison, independent of those other uses. V is separate from K because being findable and being useful when found are different things: a function word like “of” might have a K that other tokens largely skip, but its V carries the relational signal it encodes between what precedes and follows it.

Relevance between two tokens is a dot product of one’s Q against the other’s K. The dot product is high when the two vectors are aligned — the model learns through training that aligned Q/K pairs correspond to semantically relevant neighbors.

Concrete example

Sentence: “The river bank was steep”

For the token “bank” at position 3 (combined embedding [1.15, -0.55, 0.89, 1.45, ...]):

Step 1: Project to Q, K, V (per attention head, 64-dim each)
  Q_bank = embedding × W_Q = [0.32,  0.91, -0.15, ...]
  K_bank = embedding × W_K = [0.45,  0.12,  0.88, ...]
  V_bank = embedding × W_V = [0.71, -0.33,  0.52, ...]

Step 2: Dot product of Q_bank with every K in the sequence
  Q_bank · K_The   =  4.0
  Q_bank · K_river = 24.0   ← high
  Q_bank · K_was   =  3.0
  Q_bank · K_steep = 20.0   ← high

Step 3: Scale by √d_k (√64 = 8)
  Dot products grow with vector dimension — in a 64-dim space, even moderately
  aligned vectors can produce large scores. Large scores push softmax toward
  near-zero weights for everyone except the top token, destroying the gradations.
  Dividing by √d_k keeps the values in a range where softmax is informative.

  [4.0/8, 24.0/8, 3.0/8, 20.0/8] = [0.50, 3.00, 0.38, 2.50]

Step 4: Softmax converts scores to weights that sum to 1
  softmax(x_i) = e^(x_i) / Σ e^(x_j)

  The exponential amplifies differences: a score of 3.00 doesn't just mean
  "6× more relevant than 0.50" — after exponentiation it means "12× more weight".

  e^0.50 =  1.65
  e^3.00 = 20.09   ← dominant
  e^0.38 =  1.46
  e^2.50 = 12.18   ← significant
  sum    = 35.38

  → [1.65/35.38, 20.09/35.38, 1.46/35.38, 12.18/35.38]
  → [0.05,       0.57,        0.04,        0.34]

Step 5: Weighted sum of all V vectors
  output = 0.05×V_The + 0.57×V_river + 0.04×V_was + 0.34×V_steep
         = [0.62, -0.19, 0.71, ...]

The output vector for “bank” is now dominated by “river” (0.57) and “steep” (0.34). The geographic dimensions strengthen; the financial ones weaken.

With a different sentence — “The bank approved my loan” — “loan” and “approved” would dominate the weights, and the financial dimensions would strengthen instead.

The context-independent embedding encodes all possible meanings; attention resolves the contextual one.

Multi-head attention

A single attention pass can only produce one weighted mix per token. But a sentence carries several simultaneous relationships. “bank” might need to attend to “river” to resolve its meaning, while also tracking that it’s the grammatical subject of “was steep”. One weighted average can’t capture both at once.

The solution is to run several attention heads in parallel — typically 8 to 32 — each with its own independent Q, K, V projection matrices. Each head computes its own relevance scores and its own weighted mix. The outputs are concatenated and projected back to the original model dimension:

Head 1 output: [0.21, 0.33, ...]  (64-dim)
Head 2 output: [0.45, -0.12, ...]  (64-dim)
...
Head 8 output: [0.67, 0.91, ...]  (64-dim)

Concatenated: 512-dim
× W_O projection → 768-dim (back to model dimension)

No head is explicitly assigned a task — the specialization emerges from training, as different projection matrices learn to produce useful complementary views of the same input.

Multi-head attention is the main computational cost in transformers — it scales quadratically with sequence length (O(n²) in attention weights). This is why long-context inference is expensive and why various efficient attention variants (sliding window, linear attention, MLA) exist.

Transformer architecture

A transformer layer takes a sequence of vectors and produces a refined sequence — same length, same dimensions, but with each token’s vector updated based on what it learned from the others.

Layer structure

Each layer applies two steps in sequence:

1. Multi-head self-attention — rewrites each token’s vector based on surrounding context, using the mechanism above
2. Feedforward network (FFN) — applies a transformation to each token independently, not mixing information across tokens

After each step, the input to that step is added back to its output. This is called a residual connection:

output = step(input) + input

The reason: if attention or the FFN produces something unhelpful for a given token, the original vector survives anyway. It also means the gradient during training can flow backward cleanly through many layers, which makes it possible to train very deep networks without the values collapsing or exploding.

After the residual addition, layer normalization rescales the values so no single dimension drifts to an extreme magnitude through repeated passes.

Tracing “bank” through one layer:

Input:              [1.15, -0.55, 0.89, 1.45, ...]

→ Self-attention:   [0.32, -0.18, 1.12, 0.98, ...]
→ Add residual:     [1.47, -0.73, 2.01, 2.43, ...]
→ Layer norm:       [0.89, -0.44, 1.21, 1.47, ...]

→ FFN:              [0.21,  0.15, -0.33, 0.67, ...]
→ Add residual:     [1.10, -0.29,  0.88, 2.14, ...]
→ Layer norm:       [0.73, -0.19,  0.58, 1.42, ...]

Output (input to next layer):  [0.73, -0.19, 0.58, 1.42, ...]

How the FFN works: it expands each token’s vector to a much wider intermediate dimension (typically 4× the model dimension — 768 → 3,072 in BERT-base, 12,288 → 49,152 in GPT-3), applies a nonlinearity, then projects back down:

FFN(x) = W₂ × GELU(W₁x + b₁) + b₂

The GELU nonlinearity selects which dimensions are “active”. The expansion-and-compression is where most of the model’s factual knowledge is stored — interpretability research has found individual FFN layers activate for specific facts (Paris → France, water → liquid). Attention routes information between tokens; the FFN transforms it.

This repeats for each layer — 12 in BERT-base, up to 96 in large GPT-class models. Early layers tend to resolve surface patterns (word order, punctuation); later layers tend to resolve semantics (what a word means in context, how clauses relate).

The three architecture families

The original transformer (2017) had both an encoder and a decoder. Two simpler variants emerged and now dominate:

Family	How it reads the sequence	Typical use	Examples
Encoder-only	All tokens at once (bidirectional)	Classification, embeddings, NER	BERT, RoBERTa
Decoder-only	Left to right only (causal)	Text generation	GPT, LLaMA, Mistral
Encoder-decoder	Encoder is bidirectional, decoder causal	Translation, summarization (seq2seq)	T5, BART

Encoder-only models read the entire input at once — when processing a token, they can attend to everything before and after it. This is useful for tasks where you need to understand the full input before producing an answer, but it means you can’t generate text token-by-token: you’d need to attend to tokens that haven’t been produced yet.

Decoder-only models enforce a rule: each token can only attend to the tokens before it, never after. This matches what generation requires — when predicting the next token, you can only have seen what’s already been written. Each output token is generated from scratch given only the previous context, and the process repeats until the model produces an end token. Nearly all modern generative LLMs use this design.

Encoder-decoder models use both: the encoder reads the full input bidirectionally and produces a set of context vectors; the decoder generates output token-by-token and is allowed to attend to the encoder’s context vectors at each step (via a third sublayer called cross-attention). This is the natural shape for tasks with a distinct input and output, like translation.

RAG and vector databases

A model’s weights encode whatever was in its training data, frozen at the training cutoff. There’s no mechanism to update that knowledge without retraining. Retrieval-Augmented Generation works around this by fetching relevant information at query time and inserting it into the context — the model reads it like text, not memory.

How context vectors work

After passing through all transformer layers, a token’s vector is no longer its context-independent embedding — it’s been rewritten by every attention and feedforward step. Two occurrences of “bank” in different sentences produce different final vectors because they attended to different neighbors.

For RAG, what gets stored in the vector database is usually not the full token-level output but a single vector representing the whole passage — either the final-layer output of a special [CLS] token (BERT-style, trained to summarize the sequence), or the mean of all token vectors. This is called the passage embedding.

In practice, RAG systems use a dedicated embedding model optimized for semantic similarity — not the same model used for generation. A 10,000-document corpus would first be split into chunks (typically 200–500 tokens each), producing 50,000–100,000 passage embeddings. Two passages on the same topic produce similar embeddings because they use words in similar contexts, generating similar attention patterns, which produce similar final representations.

Retrieval process

1. Corpus ingestion:
   Each passage → tokenize → embedding model → passage vector
   Store passage vectors in vector database

2. Query processing:
   User query → same embedding model → query vector

3. Retrieval:
   Find the passage vectors most similar to the query vector
   using cosine similarity or approximate nearest-neighbor search

4. Augmentation:
   Retrieve top-k most similar passages
   Prepend them to the prompt → send to generation model

Cosine similarity

Two vectors are similar when they have high values at the same dimensions and low values at the same dimensions. Cosine similarity measures that alignment:

cosine(a, b) = (a · b) / (||a|| × ||b||)

The numerator a · b is the sum of elementwise products. Dividing by the magnitudes (||a|| and ||b||) normalizes for length — two vectors where one is exactly twice as long as the other still get similarity 1.0. Only direction matters, not scale.

Query vector:     [0.91, 0.03, 0.92, 0.73, 0.14, 0.88, ...]
Corpus vector A:  [0.89, 0.05, 0.90, 0.71, 0.12, 0.85, ...]  → similarity: 0.98
Corpus vector B:  [0.18, 0.81, 0.31, 0.29, 0.94, 0.22, ...]  → similarity: 0.43

Vector A has high values at the same dimensions as the query (dim1, dim3, dim4, dim6) and low values at the others — high similarity. Vector B is active on different dimensions — low similarity. Passage A is retrieved; B is not.

What RAG solves

The tradeoff is real: retrieval quality depends on embedding quality, and if retrieved context conflicts with what the model learned during training, the model may not correctly prioritize the retrieved text. RAG is not a clean override — it adds information to the context window and the model decides what to do with it.

---

This document was produced by Claude based on David Morel’s instructions and questions, drawing on the original “Attention Is All You Need” (Vaswani et al., 2017), the RoPE paper (Su et al., 2021), and the BERT paper (Devlin et al., 2018). Verify against primary sources for implementation details.