How Attention mechanism Works

When processing a sequence of data—whether words in a sentence or pixels in an image—the attention mechanism first creates a "query" representation for each element that needs context. For example, when translating the word "bank," the query encodes what the system needs to know: is this about rivers or money? These queries act like search terms in a database lookup.

The query is generated by multiplying the current element's representation by a learned weight matrix, transforming it into a form optimized for comparison. This mathematical transformation isn't random—through training on millions of examples, the network learns exactly what kinds of questions to ask for each type of input. The query captures not just "what is this element?" but "what do I need to know from other elements to understand this one?"

In the groundbreaking Transformer architecture, each position in the input generates its own unique query simultaneously. This parallel processing means that the word "bank" creates a different query than the word "river," even though they're in the same sentence. The system essentially asks multiple questions at once, with each element seeking the specific contextual information it needs.

Once queries are formed, the system compares them against "keys"—companion representations of all available input elements. This comparison uses dot product multiplication, a mathematical operation that produces higher numbers when two vectors point in similar directions. If the query for "bank" and the key for "river" have aligned representations (learned during training), their dot product yields a high score indicating relevance.

The scoring mechanism essentially measures compatibility between what one element needs (the query) and what another element offers (the key). These raw scores can vary wildly in magnitude, so the system typically divides them by the square root of the key dimension size—a normalization trick that prevents extremely large values from destabilizing later computations. Without this scaling, the attention pattern could become too sharp or too diffuse.

Every query is scored against every key in the input sequence, creating a matrix of compatibility scores. For a 10-word sentence, this means 100 scores (10 queries × 10 keys). This exhaustive comparison ensures no potentially relevant connection is missed, though it also explains why attention mechanisms become computationally expensive for very long sequences.

The raw compatibility scores cannot be used directly because they exist on arbitrary scales—one might be 0.3 while another is 47.2. The system applies the softmax function, which exponentiates each score and then divides by the sum of all exponentiated scores. This mathematical transformation converts any set of numbers into a probability distribution: all values become positive, range between 0 and 1, and sum exactly to 1.0.

These normalized values are the actual "attention weights" that determine focus allocation. If the word "bank" has weights of 0.6 for "river," 0.3 for "flows," and 0.1 for "quickly," the system will primarily attend to "river." The softmax function naturally amplifies differences: if "river" had a raw score only slightly higher than "quickly," the softmax makes that advantage more pronounced, creating clearer focus patterns.

This weighing step embodies the selectivity that makes attention powerful. Rather than treating all context equally—as older recurrent networks did—the mechanism explicitly chooses to emphasize some inputs over others. The weights are entirely learned from data; the network discovers through training which elements deserve focus for each task.

Each input element has an associated "value" representation—the actual information content that will be extracted. While queries ask questions and keys answer "I'm relevant," values contain the substantive features that get passed forward. Using our translation example, the value for "river" might encode semantic features like [WATER], [NATURE], [FLOWS], which help disambiguate "bank."

The extraction works through multiplication: each value vector is multiplied by its corresponding attention weight from the previous step. If "river" received weight 0.6, its entire value vector gets multiplied by 0.6, while "quickly" with weight 0.1 contributes only 10% of its information. This weighted selection is element-wise and happens in parallel across all inputs.

This mechanism allows relevant information to pass through at full strength while irrelevant details are mathematically suppressed. The system doesn't discard anything completely—even low-weighted elements contribute slightly—but the proportional scaling means important features dominate. The values themselves are learned representations, so the network discovers through training what information is worth encoding and extracting for each element.

The final step performs a weighted sum of all the extracted value vectors. For the word "bank" attending to a 10-word sentence, this means adding together 10 scaled value vectors (one from each word, each multiplied by its attention weight). The result is a single vector that represents "bank enriched with relevant context." Because "river" had the highest weight, its features dominate this combined representation.

This combination produces context-dependent embeddings where the same word can have different representations depending on surroundings. The word "bank" generates one combined vector in "river bank" and a completely different one in "savings bank," even though it started with the same initial representation. The attention weights determined which contextual features got mixed in.

Modern systems like GPT and BERT apply this entire query-score-weigh-extract-combine process multiple times in parallel through "multi-head attention." Each head learns to focus on different types of relationships—one might specialize in syntactic dependencies while another captures semantic similarity. The outputs from all heads are concatenated and transformed, creating rich representations that capture multiple aspects of context simultaneously. This parallel processing of different attention patterns is what gives Transformers their remarkable ability to understand nuanced language.