Transformers: attention, step by step

A transformer repeatedly asks one useful question: which parts of the current sequence matter for interpreting this token? Attention supplies the answer, but attention is only one component inside a carefully structured block.

This guide builds the transformer from its smallest useful pieces. We will start with tokens and embeddings, construct queries, keys, and values, then add multi-head attention, residual connections, normalization, and the feed-forward network.

EmbedTurn token identities into vectors.
AttendMix information across positions.
TransformProcess each position independently.
RepeatBuild increasingly useful representations.

A token needs context

Text is first split into tokens. Each token ID selects an embedding vector from a learned table. Positional information is added so the model can distinguish the first token from the fifth and understand that sequence order matters.

An embedding alone is static. The vector for bank starts the same whether the sentence describes a river or a financial institution. Contextualization changes that. After several transformer blocks, the representation depends on the surrounding sequence.

The transformation is gradual. Early layers often capture local or surface relationships. Later layers can represent broader syntax, concepts, and task-specific evidence. No single dimension has to correspond to one human-readable feature.

Queries, keys, and values

Every token representation is multiplied by three learned matrices. This produces a query, a key, and a value.

  • Query: what information is this position looking for?
  • Key: what information could this position provide?
  • Value: what content should be transferred if the match is useful?

The query from one token is compared with the keys from all allowed tokens. A dot product produces a compatibility score. Scaling keeps large vectors from pushing the softmax into extreme values. Softmax turns the scores into non-negative weights that sum to one.

scores = (queries @ keys.transpose()) / sqrt(head_dimension)
weights = softmax(scores)
context = weights @ values

The result is a weighted mixture of value vectors. A token can gather strong information from one position, distribute attention across several positions, or mostly preserve information from itself.

Inspect an attention pattern

Illustrative model

Select a token, then change the softmax temperature. Lower values concentrate attention. Higher values distribute it more evenly.

1.0

The numbers are an explanatory simulation. They do not come from a trained language model.

Attention is selective communication

It is helpful to think of attention as a temporary communication network. Each position decides how much information to collect from the positions it is allowed to see.

Why use multiple heads?

A single attention operation produces one pattern of communication. Multi-head attention runs several smaller attention operations in parallel, each with its own projection matrices. The outputs are concatenated and projected back to the model dimension.

Different heads can specialize in different relationships, but they are not guaranteed to have a simple interpretation. One head may favor nearby tokens, another may track a repeated entity, and another may carry broader contextual information. Their combined output is more expressive than a single large attention map.

More heads do not automatically mean better reasoning. The total model dimension is divided across the heads. Architecture design balances the number of heads, the dimension per head, memory use, and the needs of the training objective.

Masking controls the information flow

An encoder normally allows every token to attend to every other token. A decoder used for next-token prediction applies a causal mask. Position five can use positions one through five, but it cannot inspect position six because that token belongs to the future.

The mask is applied before softmax by assigning forbidden connections a very negative score. Their probability becomes effectively zero. This small operation creates the autoregressive behavior used by language models.

Other masks handle padding, missing modalities, or application-specific constraints. In multimodal systems, masking can also control which visual and textual tokens communicate at different stages.

The rest of the transformer block

Attention moves information between positions. A feed-forward network then transforms the representation at each position independently. It usually expands the vector into a larger hidden dimension, applies a nonlinearity, and projects it back down.

Residual connections carry the input around both sublayers. Instead of forcing each component to reconstruct everything, the component learns a useful update. Normalization keeps activations in a workable range and supports stable optimization.

x = x + attention(normalize(x))
x = x + feed_forward(normalize(x))

This pre-normalization form is common in modern large models. The exact order, activation function, positional method, and normalization type vary, but the high-level pattern remains recognizable.

Where visual tokens enter

A language model does not require every input token to begin as text. If a projector maps visual features into the correct embedding dimension, those vectors can be placed before or alongside the question tokens.

Self-attention then creates communication paths between modalities. A question token can attend to visual tokens, and later textual representations can carry forward the visual evidence needed to generate an answer. Some architectures instead use explicit cross-attention blocks, but the principle is similar: learned attention controls information exchange.

This makes the quality of the visual prefix important. If those representations are noisy or poorly aligned, attention can faithfully distribute unreliable information. The transformer cannot recover visual evidence that never survived the encoder and projector.

An attention map is not a complete explanation

Attention weights show one route of information mixing. Residual paths, value projections, feed-forward layers, and later blocks also affect the output. Treat a heatmap as evidence about model behavior, not a full causal account.

The computational tradeoff

Full self-attention compares every token with every other token. If the sequence length doubles, the score matrix contains roughly four times as many entries. This quadratic growth matters for long documents, high-resolution image patches, and long multimodal contexts.

Several families of methods reduce this cost through sparse patterns, local windows, compressed representations, or more efficient attention kernels. The engineering details change, but the original transformer remains the clearest foundation for understanding them.

A useful mental model

A transformer block alternates between communication and computation. Attention lets positions exchange information. The feed-forward network transforms what each position has learned. Residual connections preserve an information highway through the stack.

Queries, keys, and values are not separate databases or fixed semantic roles. They are learned views of the same token representations. Their job is to create a differentiable routing system that the model can adapt to its training task.

Further reading

Continue learning