Positional Embeddings

Why Attention Needs Position

Compare dog bites man with man bites dog. The two sequences contain the same token identities, but order reverses who acts on whom. A transformer therefore needs token identity from the embedding matrix and positional information that distinguishes the arrangements.

Self-attention without positional information is permutation equivariant: if we permute the input rows, the output rows undergo the same permutation. Queries and keys can match content, but their dot product contains no general measure of which token came first or how far apart two tokens are. A causal mask gives decoder-only models an asymmetry by hiding future positions. The set of visible positions changes across the sequence, but the mask alone does not supply a reusable representation of absolute index or pairwise distance.

Positional Encoding: A positional encoding is any mechanism that makes a transformer's computation depend on absolute position, relative offset, or both.

“Positional embedding” is often used as an umbrella term, but not every method performs an embedding lookup. Learned absolute vectors are embeddings. Fixed sinusoidal vectors are encodings. Rotary Position Embedding (RoPE) transforms queries and keys, while Attention with Linear Biases (ALiBi) adds a distance penalty to attention logits. The distinctions matter in mechanistic interpretability because each method inserts position into a different part of the computation.

Absolute Position Vectors

The most direct approach assigns a vector $\mathbf{p}_i \in \mathbb{R}^{d_{\text{model}}}$ to absolute position $i$ and adds it to the token embedding:

$$\mathbf{r}^0_i = W_E[t_i,:] + \mathbf{p}_i.$$

With learned absolute positional embeddings, the vectors form a matrix $W_P \in \mathbb{R}^{n_{\text{ctx}} \times d_{\text{model}}}$. Training learns one row for each supported position. Token identity and position are separate additive contributions at the input, but later attention and multilayer perceptron (MLP) blocks read their sum and can respond to interactions between them.

A learned table has a fixed set of rows. Positions beyond the table cannot be represented without resizing or replacing it, and rows that were never trained do not acquire useful behavior automatically. Even positions inside the table may generalize poorly if training rarely placed relevant patterns there.

Pause and think: Can the model separate identity from position?

If the initial state is $\mathbf{e}_t+\mathbf{p}_i$, have we permanently lost which part came from the token and which part came from its position?

The addition does not preserve labeled slots, but learned downstream directions can respond mostly to token subspaces, position subspaces, or combinations of both. Whether the trained model keeps them cleanly separable is an empirical property, not a guarantee of addition.

Fixed Sinusoidal Encodings

The original transformer replaced the learned position table with a deterministic pattern of sine and cosine waves [1]Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., et al.
NeurIPS, 2017. For coordinate pair $2k,2k+1$,

$$\text{PE}(i,2k)=\sin\left(i/10000^{2k/d_{\text{model}}}\right),$$

$$\text{PE}(i,2k+1)=\cos\left(i/10000^{2k/d_{\text{model}}}\right).$$

Fixed Sinusoidal Positional Encoding: Position $i$ is represented by sine-cosine pairs with fixed angular frequencies $\omega_k=10000^{-2k/d_{\text{model}}}$. The model learns how to read these vectors, but it does not learn the vectors themselves.

Each pair is a point on a unit circle:

$$\bigl[\sin(i\omega_k),\ \cos(i\omega_k)\bigr].$$

Fast pairs wind around the circle many times over a sequence, while slow pairs change only slightly. Taken together, the pairs give each position a multiscale phase signature. The interactive diagram shows one pair at a time for an eight-dimensional toy encoding. Pair $k=0$ changes fastest; increasing $k$ increases its wavelength.

Scroll the diagram horizontally to compare both views.

Position i

18

Coordinate pair k

1

Position 18, pair 1.

A position shift $\Delta$ rotates every pair by a known angle $\Delta\omega_k$:

$$\begin{bmatrix}\sin(i\omega_k)&\cos(i\omega_k)\end{bmatrix} \begin{bmatrix} \cos(\Delta\omega_k)&-\sin(\Delta\omega_k)\\ \sin(\Delta\omega_k)&\cos(\Delta\omega_k) \end{bmatrix} = \begin{bmatrix}\sin((i+\Delta)\omega_k)&\cos((i+\Delta)\omega_k)\end{bmatrix}.$$

This exact linear relationship gives downstream layers a structured way to compare offsets. The encoding is still added to the token embedding as an absolute position vector. A learned attention head must use its projections to turn those phases into whatever relative-position computation it needs.

The formula produces vectors for positions beyond the training length, unlike a finite learned table. Defined inputs do not guarantee valid extrapolation: the model's attention patterns and learned algorithms were optimized on the lengths it saw during training.

Rotary Position Embedding

Rotary Position Embedding applies position-dependent rotations after the query and key projections [2]RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., et al.
arXiv, 2021. Split each query and key into pairs of coordinates. At position $i$, rotate each pair by an angle proportional to $i$, using a different angular frequency for each pair:

Rotary Position Embedding (RoPE): RoPE rotates query and key coordinate pairs by position-dependent angles, causing their dot product to depend on the relative offset between their positions.

$$\widetilde{\mathbf{q}}_i=\mathbf{q}_i R_i, \qquad \widetilde{\mathbf{k}}_j=\mathbf{k}_j R_j.$$

The matrices $R_i$ are block-diagonal rotations. Because rotations compose by adding their angles, the attention dot product becomes

$$\widetilde{\mathbf{q}}_i\widetilde{\mathbf{k}}_j^T = \mathbf{q}_i R_i R_j^T \mathbf{k}_j^T,$$

and $R_iR_j^T$ depends on the relative offset $i-j$. RoPE therefore starts with absolute indices but exposes their difference inside the query-key interaction.

The interactive diagram isolates this relative effect in one two-dimensional coordinate pair. It fixes the unrotated content vectors to $\mathbf{q}=\mathbf{k}=(1,0)$, rotates them to positions $i$ and $j$, and plots their dot product against the offset $\Delta=i-j$. Real heads use different content vectors and many frequency pairs, but the same relative-rotation identity applies to each pair.

Scroll the diagram horizontally to compare both views.

Query position i

10

Key position j

4

Angular frequency ω

0.35

Relative offset 6.

RoPE preserves the norm of each rotated query and key. It changes their alignment, which changes attention scores, without writing a position vector into the residual stream. The name can otherwise be misleading: in the standard use of RoPE, the token embedding itself is not rotated, and the value vectors are not position-rotated by this operation.

The rotation frequencies determine how quickly each coordinate pair changes with distance. Slow pairs carry coarse, long-range variation; fast pairs distinguish nearby offsets but wrap around more quickly. Methods that rescale positions or frequencies can extend a RoPE model's usable context, but extension remains an out-of-distribution intervention whose quality must be measured.

Pause and think: Where would you patch RoPE?

Suppose we want to preserve a token's content but replace the positional effect on one attention head. Should we patch the residual stream before the head?

Not necessarily. Standard RoPE enters after the head computes queries and keys. Patching the pre-head residual stream mixes content with every downstream use of that state. Patching the rotated query or key isolates a narrower positional site, although it also changes the query-key interaction and must be interpreted as a model-internal counterfactual.

Relative Position Inside Attention

Many tasks depend more directly on offsets than absolute indices. “Attend to the previous token” describes relative offset $-1$ at every destination position. A learned absolute scheme must infer that repeated relation from pairs of absolute vectors, while a relative position representation can place the offset directly into attention.

One family adds learned relative terms to the key or value used for a pair of positions. The attention score can then depend on a vector indexed by $i-j$, and the information moved can also vary with that offset [3]Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., Vaswani, A.
NAACL, 2018. Another family adds a scalar bias $b_{i-j}$ to the score:

$$s_{ij}=\frac{\mathbf{q}_i\mathbf{k}_j^T}{\sqrt{d_k}}+b_{i-j}.$$

This separates a content-dependent term from a position-dependent preference before softmax. The separation is visible in the equation, but the final attention probability still couples them through normalization against every eligible source position.

ALiBi uses a particularly simple fixed bias. For causal attention, head $h$ receives a negative penalty proportional to backward distance:

$$s^{(h)}_{ij}=\frac{\mathbf{q}_i\mathbf{k}_j^T}{\sqrt{d_k}}-m_h(i-j), \qquad j\leq i,$$

where slope $m_h>0$ differs across heads. ALiBi does not add vectors to token representations. It gives each head a distance-dependent recency preference directly in its attention logits and was designed to improve extrapolation beyond training sequence lengths [4]Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., Lewis, M.
ICLR, 2022.

Comparing the Main Families

Method	Where position enters	Signal represented	Consequence for analysis
Learned absolute embedding	Added to the initial residual stream	Absolute index	Position can flow through attention, MLP, and direct residual paths
Fixed sinusoidal encoding	Added to the initial residual stream	Absolute index with structured frequencies	Position begins as an additive input, with exact algebraic relations between offsets
Learned relative representation	Attention keys, values, or logits	Usually a bucketed or clipped offset	The attention operation contains an explicit pairwise positional term
RoPE	Rotations of queries and keys	Absolute phase whose dot product depends on relative offset	Query-key circuits vary with source-destination offset
ALiBi	Additive attention-logit bias	Linear distance penalty per head	Content and distance add before softmax; no positional write enters the residual stream

No row in the table guarantees length extrapolation. A model learns computations under a training distribution of lengths and positions. A mathematically defined encoding at position 100,000 only supplies an input there; it does not prove that attention patterns, MLP behavior, or learned algorithms remain valid.

Why Position Changes Interpretability

An attention pattern combines content and position. A head that attends to the previous token may implement a positional rule, a content match that usually occurs one token back, or both. Inspecting the pattern alone cannot distinguish these mechanisms. We can compare prompts that preserve content while changing offsets, inspect the query-key score decomposition, or intervene on the positional term.

RoPE makes a head's query-key circuit a family of maps indexed by relative offset. If two feature directions align after a short-distance rotation but not a long-distance rotation, the same content pair can receive different scores at different separations. Collapsing those scores into one token-to-token matrix discards part of the mechanism.

Absolute embeddings create a different complication for activation patching. Moving a cached activation from position $i$ to position $j$ transfers content that was computed with $i$'s positional contribution. That may be the intended intervention, or it may create an inconsistent state. Position-aligned controls and offset-preserving prompt pairs help separate the two interpretations.

Special positions can also become computational anchors. Beginning-of-sequence tokens, separators, and repeated formatting positions combine token identity with predictable location, so a circuit may use them as attention sinks or reference points. Calling such behavior “positional” does not imply that a dedicated positional vector alone causes it.

Looking Ahead

The Attention Mechanism builds queries, keys, values, masks, and softmax into the full information-routing operation. Positional methods specify where order enters that operation, which lets us state more precisely what an attention head reads and why it prefers one source position over another.