How AI Knows What "It" Means: Attention Explained

Read this sentence:

The cat sat on the mat because it was tired.

What does “it” mean? The cat, obviously. Mats don’t get tired.

Now change one word:

The cat sat on the mat because it was dirty.

Now “it” is the mat. Cats can be dirty too, so the sentence is a little ambiguous, but most people read it as the mat that needs cleaning. You flipped the meaning of “it” without noticing you did anything.

For a computer, that flip is hard. Nothing about the letters i-t tells you whether it points at a cat or a mat. The answer lives in the other words, including one that comes later.

The trick modern AI uses to solve this is called attention. It’s the core idea of the 2017 paper “Attention Is All You Need”, which introduced the Transformer, the T in GPT. This post explains it in plain English, with no maths beyond “multiply and add” and percentages.

This is part 4 of How AI Works, a five-part series that follows one sentence through a language model.

Where we left off

In part 3, we turned words into numbers. The model first splits your text into tokens (pieces of words, covered in part 2), looks up each token’s token ID, and swaps each ID for an embedding: a long list of numbers, called a vector, that places the token on a kind of map of meaning. Tokens with similar meanings end up close together on that map.

But part 3 ended on a problem. The embedding for a token is looked up from a fixed table. “Bank” gets the same starting numbers in “river bank” and “bank account”. And “it” gets the same numbers whether it means a cat, a mat or a tax refund. On its own, the embedding for “it” means almost nothing.

So the model needs a way for each token to update its numbers based on the tokens around it. That’s what attention does.

(In our example sentence each word happens to be one token, so I’ll say “word” and “token” interchangeably here. The model itself always works on tokens.)

Before transformers: reading through a keyhole

Language models existed long before 2017. The most common kind for text was the recurrent neural network, or RNN.

An RNN reads text the way you’d read through a keyhole: one word at a time, left to right. It carries a small “memory” (a vector of numbers) from each word to the next, updating it as it goes. By the time it reaches “it”, everything it knows about “The cat sat on the mat because” has been squeezed into that one running memory.

That design had two big problems.

1. Long distances were hard. Information about an early word has to survive many update steps to reach a later word, and it tends to fade. Improved versions like LSTMs helped, but as Google’s 2017 blog post on the Transformer put it, the more steps a decision needs, the harder it is for a recurrent network to learn.
2. It was slow to train. Word 5 can’t be processed until word 4 is done. The Transformer paper says this “sequential nature precludes parallelization”: chips built to do thousands of calculations at once sat mostly waiting.

Attention itself wasn’t new in 2017. In 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio added it to an RNN translation system: instead of squeezing a whole sentence into one fixed-size memory, the model could look back over all the source words and pick out the relevant ones for each word it translated.

The 2017 paper by Ashish Vaswani and seven Google colleagues went further. They dropped the recurrent part completely and built the model around attention. No keyhole: every word looks at the other words directly, and all words are processed at once. Google reported that it trained up to an order of magnitude faster than the alternatives it was tested against, and translated better.

Attention in one sentence

Here’s the whole idea:

Each token looks at the other tokens, scores how relevant each one is, and pulls in a blend of their meaning, weighted by those scores.

For “it”, that means: notice that “cat” and “tired” matter a lot and “the” and “on” barely matter, and shift its own numbers toward “a tired cat”. After attention, “it” is no longer a blank pronoun.

Every token does this. “Bank” finds “river” and shifts toward water, or finds “account” and shifts toward money. Same starting numbers, different meaning, decided by the company the word keeps.

The interesting question is how a token decides what’s relevant.

Questions, labels and contents: query, key and value

For each token, the model makes three new vectors from its embedding. It does this by multiplying the embedding by three tables of numbers (called weight matrices) that the model learned during training. The three results are called the query, the key and the value.

A helpful way to picture them is a room full of people at a meeting, one person per word.

• Query: the question a token is asking. For “it”, roughly: “I’m a pronoun. Which thing am I standing in for?”
• Key: the label a token is wearing. For “cat”, roughly: “I’m a noun. I’m a living thing that can be tired.” For “the”, roughly: “I’m just a ‘the’.”
• Value: what a token hands over if you listen to it. The actual information to pass along.

To decide who to listen to, “it” holds up its question and compares it with every label in the room. A good match gives a high score. A poor match gives a low score. Then “it” collects the values, taking a big share from the good matches and a tiny share from the poor ones.

Where the analogy breaks. Nobody writes these questions and labels, and they aren’t sentences. They’re lists of numbers, and “I’m a pronoun” is my translation of a pattern, not something you can read off them. Like embeddings, the tables that make queries, keys and values start out random and get adjusted during training until the model gets good at predicting the next token. “Which noun does ‘it’ point to?” is a pattern the model may discover. Nobody programs it in.

From scores to weights: softmax

How do you “compare a question with a label” when both are lists of numbers? You multiply them together position by position and add up the results. This is called a dot product. If the two lists tend to be large in the same places, the total is large: a good match. If they don’t line up, the total is small or negative.

(The paper also divides each score by a fixed number so the scores don’t get too large. You can safely forget that detail.)

Raw scores can be negative and don’t add up to anything in particular. So the model runs them through a function called softmax, which turns any list of scores into weights that are all positive and add up to 1 (or 100%, if you prefer percentages).

Softmax does two things:

1. It makes every score positive, in a way that stretches the gaps: a slightly higher score becomes a much bigger number. (Technically it raises the number e, about 2.718, to the power of each score.)
2. It divides each of those by their total, so they add up to 1.

A tiny worked example with three words and made-up scores:

A score gap of just 1 between “cat” and “mat” turns into a weight of 70% versus 26%. That’s the stretching at work.

The weighted mix

Now the payoff. The new information for “it” is a weighted mix of everyone’s values: 70% of the cat’s value, plus 26% of the mat’s value, plus 4% of the sat’s value, added up position by position. That mix is added to the vector “it” already had, so “it” keeps its own meaning and gains context on top.

Here’s what a pattern like that might look like across the full sentence, for both versions. These numbers are illustrative. I made them up to show the shape of the idea; they are not output from a real model. In this picture “it” can see the whole sentence, as in the original 2017 translation model, which read the full source sentence before translating. (Chatbots work a little differently. More on that below.)

Change one word at the end, and most of the attention moves from “cat” to “mat”. The vector for “it” ends up leaning toward whichever noun fits.

Researchers really can draw pictures like this from real models; the 2017 paper shows heads that appear to link pronouns to their nouns. But real patterns are messier than my table, and reading meaning into them is not an exact science.

Many heads: several questions at once

One round of attention asks one kind of question. But a word has many relationships at once. “It” needs to know what it refers to. “Sat” needs to know who did the sitting.

So the Transformer runs several attention calculations side by side, each with its own query, key and value tables. Each one is called a head, and different heads end up looking for different things. A 2019 study of Google’s BERT model by Kevin Clark and colleagues found heads that link verbs to their objects, nouns to words like “the”, and pronouns to what they refer to. Others look at a fixed position, such as the next word. Many do things that are hard to put into words at all. The results from all heads are combined into one updated vector per token.

And it doesn’t happen just once. Attention is stacked in layers, each working on the richer vectors the previous layer produced. The original 2017 model had 8 heads and 6 layers on each side. The largest GPT-3 model, from 2020, had 96 layers with 96 heads each. Part 5 walks through one layer in detail.

The chatbot rule: no peeking ahead

Chatbots like ChatGPT, Claude and Gemini are built for next-token prediction: given the tokens so far, give a probability for every possible next token, pick one, add it, repeat. (Part 1 covers this loop.) To learn that, the model must never see the answer. When it’s trained to predict the word after “because”, it can’t be allowed to look at “it was tired”.

So these models use causal masking: each token may only attend to itself and the tokens before it. Scores for later tokens are blocked out before softmax, so they get a weight of exactly zero. The 2017 paper describes the same trick in its decoder, the half of the model that writes the translation, to “prevent positions from attending to subsequent positions”.

Look at the highlighted row. In a chatbot, when the model processes “it”, the deciding word “tired” hasn’t been seen yet. At that point “it” honestly can’t know whether it’s the cat or the mat. A sensible pattern would be to spread attention across both candidates.

So how does the model get it right? Later tokens do the work. When “tired” is processed, it can look back at “it”, “cat” and “mat”, and across the layers the vectors for “was” and “tired” can pick up “the thing that’s tired is the cat”. The question gets answered, just at a later position than in my table, which shows the full two-way view a translation model’s encoder has.

One point that often confuses people: masking doesn’t mean the model reads your prompt word by word like an RNN. All the tokens of your message are processed in parallel, each blocked from seeing the ones after it. Only the reply is produced one token at a time.

The catch: the cost grows with the square

Attention has a price. If every token looks at every other token, the number of comparisons grows with the square of the length:

Causal masking roughly halves that, but the shape stays: double the length, and the attention work goes up about four times. The 2017 paper says this openly: a self-attention layer’s cost grows with the length squared, a recurrent layer’s only in proportion to it. The trade was deliberate: more total work, but work that can happen all at once on modern chips.

Why does this matter to you?

• It’s a big reason context windows have limits. The context window is the maximum number of tokens a model can look at at once (see part 2). Every extra token is compared with all the others, in every head and layer.
• Long inputs are slower and cost more. APIs charge per token, and a huge paste slows every reply. More on those costs in The Hidden Costs of AI Tools Nobody Shows in the Demo.
• Long inputs are used unevenly. A 2023 study by Nelson Liu and colleagues, “Lost in the Middle”, found that models answered best when the needed information was at the start or end of a long input, and noticeably worse when it was buried in the middle. That held even for models built for long contexts. The exact reasons are still researched, so treat “too many tokens competing for attention” as an intuition, not a proven explanation.

Engineers keep softening the cost. FlashAttention (Tri Dao and colleagues, 2022) computes exactly the same result while using memory far more cleverly; other methods approximate attention or limit which tokens each one looks at. But the basic trade-off remains: more tokens means more attention to share around.

What this means when you use AI

You don’t need to know any of the maths to use this. Three habits follow directly from how attention works.

1. Name things instead of saying “it”. In a long message about two invoices and a customer, “it” could point anywhere. “The March invoice” can’t. You’re making the attention job easy.
2. Put the important instructions where they stand out. Start with them, label them clearly, and don’t bury the one rule that matters in the middle of a long paste. My 5-part prompt recipe is built around this.
3. In long chats, restate key facts or start fresh. After a lot of back-and-forth, paste a short summary of what matters and continue from there, or open a new chat with that summary.

None of this is a magic phrase. It just helps the model’s attention land on the right words.

Recap

• A token’s embedding means the same thing everywhere. Attention lets each token update its meaning from its context.
• Each token makes a query (its question), a key (its label) and a value (what it hands over). Query-key matches give scores.
• Softmax turns the scores into weights that add up to 1, and the token takes a weighted mix of values.
• Many heads look for different relationships at once, and many layers refine the result.
• In chatbots, causal masking lets each token look only at earlier tokens. Later words help settle earlier ambiguities.
• Attention work grows with the square of the length, which is a big reason long context is expensive and limited.

Next: part 5, “Inside a Transformer: One Word’s Journey to the Next Word”, puts the pieces together and follows one token through a full layer to the model’s prediction.

Sources: Vaswani et al., “Attention Is All You Need” (NeurIPS 2017: recurrence “precludes parallelization”, scaled dot-product attention, 8 heads, 6 layers, decoder masking, per-layer cost of order n²·d in Table 1, anaphora-resolution heads in Figure 4); Bahdanau, Cho and Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate” (2014); Jakob Uszkoreit, “Transformer: A Novel Neural Network Architecture for Language Understanding” (Google Research blog, August 31, 2017: RNNs and many sequential steps, training “up to an order of magnitude” faster); Brown et al., “Language Models are Few-Shot Learners” (2020: GPT-3 175B with 96 layers and 96 heads); Clark et al., “What Does BERT Look At?” (2019); Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023); Dao et al., “FlashAttention” (2022). All attention weights in this post are illustrative, not output from a real model.

Related: What an “AI Agent” Actually Is, Without the Hype, and Stop Getting Generic Answers From AI: The 5-Part Prompt Recipe.