Kalamang is spoken by fewer than 200 people, on a few small islands off the coast of Western New Guinea in Indonesia. There is almost nothing about it on the internet. That makes it a perfect trap for an AI model: it can’t have learned Kalamang during training, because there was nothing to learn from.
What does exist is a book. A linguist, Eline Visser, spent years documenting the language and wrote A Grammar of Kalamang, about 500 pages long. Along with it there is a bilingual word list of roughly 2,000 entries and about 400 example sentences. In 2023, a group of researchers (including Visser) turned this into a test called MTOB, “Machine Translation from One Book”: give a model the materials, then ask it to translate.
In February 2024, Google published its technical report for Gemini 1.5 Pro and tried exactly that. All the Kalamang materials together came to about 250,000 tokens. Google put the whole lot into the model’s prompt. No fine-tuning, no retraining. Just: here is the book, now translate this sentence.
The results, scored 0 to 6 by a human who had learned Kalamang from the same materials:
- With no book, Gemini 1.5 Pro scored essentially zero. As expected, it didn’t know the language.
- With the full book in its prompt, it scored 5.52 translating English into Kalamang. The human learner scored about 5.6. Close to level.
- In the other direction, Kalamang into English, the model scored 4.36 against the human’s 5.52. A real gap.
So the honest headline is: for one direction of translation, a model reading one grammar book in its prompt did about as well as a person who studied the same book. The report itself says the task isn’t solved, and that the human and the model made different kinds of mistakes. The human couldn’t re-read 500 pages for every sentence. The model had the whole book “open” but applied rules inconsistently.
Still, that is a remarkable thing to watch. A quarter of a million tokens, read in one go, used well enough to learn a language. This post is about how that is even possible, what it costs, and where it breaks.
If tokens or attention are new to you, start with the basics first. This post assumes you know them.
The problem: attention gets expensive fast
Quick reminder of how attention works. For every token, the model asks: “which other tokens in this text matter to me right now?” To answer, it compares the token with every token before it.
That “every” is the problem.
With 4 tokens, there are 16 possible pairs to compare (counting both directions, to keep it simple). With 8 tokens, 64. With 1,000 tokens, a million. Double the length and the work goes up four times. This is called quadratic growth: the cost grows with the length multiplied by itself.
At 250,000 tokens, the full grid has over 60 billion cells per attention head, per layer. Early transformers were trained with a few thousand tokens of context for exactly this reason. Going longer wasn’t impossible. It was just ruinously expensive.
There are really two costs hiding here, and it helps to keep them apart:
- Compute: the number of comparisons. This grows quadratically when the model reads your prompt.
- Memory: what the model has to keep around while it works. This is where the KV cache comes in.
The KV cache: the model’s notes on everything it has read
In attention, each token produces three small lists of numbers (vectors):
- a query: “what am I looking for?”
- a key: “here is what I contain, if you’re looking”
- a value: “here is what I’ll pass on if you pick me”
When a new token arrives, its query is compared with the keys of all earlier tokens, and it takes a weighted mix of their values.
Now think about how a model writes. It produces one token at a time. Token 1,001 needs the keys and values of tokens 1 to 1,000. Token 1,002 needs those same 1,000, plus one more. Recomputing all of them for every new word would be absurdly wasteful, because they don’t change.
So the model saves them. That saved store of keys and values, for every token, in every layer, is the KV cache.
A decent analogy is a student reading a long book with a notebook. Instead of re-reading from page one every time they need to recall something, they jot a short note per page and scan the notes. The analogy breaks in one important way: the student’s notes are a summary. The KV cache is not. It keeps a full set of numbers for every single token, at every layer. Nothing is condensed.
That is why it gets big.
How big?
The vLLM paper from UC Berkeley (2023) gives a clean worked example. For a 13-billion-parameter model (Meta’s OPT-13B), one token’s KV cache takes about 800 KB: 2 (key and value) × 5,120 (numbers per vector) × 40 (layers) × 2 bytes per number. At that model’s maximum of 2,048 tokens, one conversation needs about 1.6 GB just for the cache.
Now do the simple arithmetic yourself. The same model at 128,000 tokens would need around 100 GB of cache for one user. That is more than the whole memory of most single GPUs, before you even load the model’s weights.
So the honest picture of long context is this: reading the prompt costs compute, and remembering it costs memory, for as long as the conversation is alive. That memory is what limits how many users one GPU can serve at once, and it’s a big part of why long prompts cost more.
The vLLM team also found that existing serving systems wasted a lot of this memory: in their measurements, only 20.4% to 38.2% of the reserved KV cache memory held actual token data. Their fix, PagedAttention, borrows an idea from operating systems: store the cache in small fixed-size pages instead of one big reserved block per request. They reported 2 to 4 times higher throughput than the systems they compared against.
How models stretch to long context
There isn’t one trick. There’s a toolbox, and each tool attacks a different cost. Here are the main ones, described as accurately as I can in plain words.
1. Make the cache smaller: shared keys and values
In standard attention, every “head” (think of heads as separate attention channels looking for different things) has its own keys and values. Multi-query attention lets all heads share one set. Grouped-query attention (GQA), from a 2023 Google paper, is the middle ground: heads are split into groups, and each group shares one set of keys and values. The authors reported quality close to normal attention with speed close to multi-query. Fewer key-value sets means a smaller cache. Many modern open models use GQA; Mistral 7B, for example, says so in its paper.
2. Make attention cheaper to run: FlashAttention
This one is often misunderstood, so let me be precise. FlashAttention (Tri Dao and colleagues at Stanford, 2022) does not approximate attention and does not reduce the number of comparisons. It computes exactly the same answer.
What it changes is where the work happens. A GPU has a small amount of very fast memory on the chip and a much larger amount of slower memory beside it. Standard attention writes the whole giant comparison grid out to the slow memory and reads it back. FlashAttention cuts the work into tiles that fit in the fast memory, and never stores the full grid at all.
The analogy: same recipe, same amount of chopping, but you stop walking to the pantry for every single ingredient. The paper reported about 3 times faster training for GPT-2 at 1,000 tokens, and it let models train on much longer sequences than before. The compute still grows quadratically. It’s just much less painful.
3. Don’t look at everything: sliding window and sparse attention
The other route is to stop comparing every token with every other token.
- Sliding window attention: each token only looks at the last few thousand tokens. Mistral 7B uses a window of 4,096. Information from further back can still travel, because each layer passes things forward, a bit like a rumour moving along a line of people. Mistral’s paper calculates a theoretical reach of about 131,000 tokens across its layers. It also says a fixed window lets the cache stay a fixed size, which cut cache memory 8 times on 32,000-token sequences.
- Sparse attention: tokens look at a chosen pattern of other tokens, not all of them. OpenAI’s Sparse Transformer (2019) used fixed patterns. Longformer (2020) combined a local window with a few “global” tokens that everyone can see.
The trade-off is real: these approaches are cheaper because the model does not look at some of the text directly. A theoretical reach of 131,000 tokens is not the same as reliably using a fact from 100,000 tokens back.
4. Teach the model about positions it has never seen: RoPE scaling
Attention on its own has no sense of order. Models need to be told where each token sits. A popular method is RoPE (rotary position embedding, from Su and colleagues, 2021). The idea in plain words: each query and key is rotated by an angle that depends on its position, so when two tokens are compared, the difference in their angles tells the model how far apart they are.
The problem: if a model was trained on 4,000-token texts, it has never seen the angles for position 50,000. Feed it longer text and it gets confused.
Position Interpolation (Meta, 2023) has a neat fix. Instead of pushing the model into unfamiliar positions, squeeze the long text’s positions down to fit into the range it already knows. Position 8,000 is treated a bit like position 4,000 was, position 2 like position 1, and so on. Then do a short round of extra training so the model adjusts. The paper extended LLaMA models to 32,768 tokens with fewer than 1,000 training steps. YaRN (2023) refined the idea and reported needing about 10 times fewer tokens and 2.5 times fewer training steps than earlier methods.
The analogy: you’ve learned to read a ruler marked in centimetres, and now someone hands you a ruler twice as long with the same number of marks. You can still read it, but you need a bit of practice, and fine details get harder to tell apart.
5. Spread it across many chips
For truly huge contexts, the work is split over many GPUs, each handling part of the sequence. The Gemini 1.5 report credits “major advances in training and serving infrastructure” but doesn’t spell out the exact recipe, so I won’t guess at it.
Needle in a haystack: the famous test, and what it misses
How do you check that a model can actually use a million tokens? The most famous test is needle in a haystack, popularised by Greg Kamradt in 2023. Take a long, boring text (often a pile of Paul Graham essays), hide one odd sentence somewhere in it, like “The special magic number is 42”, and ask the model for it. Repeat at every length and every position.
Gemini 1.5 Pro did extremely well. Google reported 100% recall up to 530,000 tokens and over 99.7% up to 1 million, with retrieval staying high in research tests up to 10 million tokens.
But read the same report a little further and it adds its own caveats:
- More needles, lower scores. With 100 different needles hidden at once, Gemini 1.5 Pro found around 70% of them up to 128,000 tokens, and over 60% up to 1 million. Better than GPT-4 Turbo on the same test, but far from 99.7%.
- Prompt and needle choice change the result. The authors say future versions of the test should account for that.
Other researchers pushed harder. NVIDIA’s RULER benchmark (2024) added harder variants: multiple needles, needles that look like each other, tracing a chain of facts, counting things across the whole text. Almost every model aced the basic needle test. But of 17 models that all claimed 32,000 tokens or more, only about half held up well at 32,000 on the harder tasks.
The lesson: a needle test checks one skill, finding a sentence that stands out. Real work usually needs the model to connect many facts spread across the text, none of which stand out. Those are different skills, and the second is much harder.
Lost in the middle
There is a second, stranger problem. Even when the information fits, where it sits matters.
In 2023, Nelson Liu and colleagues at Stanford gave models a question plus 20 documents (about 4,000 tokens), with the answer in just one document. Then they moved that document around. Accuracy was highest when the answer was at the start or the end, and dropped when it was in the middle: a U-shaped curve. In the worst cases, GPT-3.5-Turbo did worse with the answer document in the middle than it did with no documents at all (56.1% when answering from memory alone).
People show a similar pattern, remembering the first and last items in a list best, so it’s a familiar shape. The analogy breaks because the model isn’t tired or distracted. It’s a learned habit, probably because so much training text puts the important parts at the beginning and end.
Newer models have improved on this, and the effect varies a lot by model. But the practical advice still holds: put the most important material and your actual question near the end of a long prompt, or at the start, not buried in the middle. And test it on your own data rather than trusting a context length on a spec sheet.
Long context vs RAG: which one should you use?
If you build with AI, this is the question that matters. You have a big pile of documents. Do you paste everything into a long-context model, or use RAG (retrieval-augmented generation: search for the few relevant chunks and only send those)?
A 2024 study by Li and colleagues, published at EMNLP, compared the two head to head. Their finding, roughly: when you can afford it, long context gave better answers on average, but RAG was far cheaper. Their suggested compromise, “Self-Route”, lets the model try RAG first and only fall back to long context when it judges the retrieved chunks aren’t enough. That kept quality close to long context at a much lower cost.
Here is how I’d think about the trade-off:
Long context is better when:
- The task needs the whole thing: summarising a full contract, spotting contradictions across a report, or learning a language from a grammar book.
- You don’t know in advance which parts matter, so search would miss things.
- The document set is small enough to fit and you’ll ask many questions about it.
RAG is better when:
- The collection is huge (a whole company wiki will not fit in any context window).
- The data changes often.
- Each question only needs a few facts.
- Cost and speed matter, which is almost always.
- You want to show which source an answer came from.
Things that change the maths:
- Every token you send is paid for, on every call. A 200,000-token prompt sent 100 times a day adds up fast. I wrote more about this kind of creeping cost in the hidden costs of AI tools.
- Caching helps. Several providers, including Google and Anthropic, let you cache a repeated chunk of prompt (say, the same manual at the start of every request) so repeat calls cost less. Pricing changes often, so check the current numbers at the time you build.
- Speed. The model has to read the whole prompt before writing the first word. Longer prompts mean a longer wait.
In practice, many good systems use both: retrieval to narrow a huge pile down to a few dozen documents, and a long context window so those few dozen can be read in full instead of in tiny chunks.
What it means in practice
If you’re building with or buying AI:
- Treat the advertised context length as a ceiling, not a promise. “1 million tokens” means the model accepts that much. It doesn’t mean it reasons equally well over all of it.
- Test with your own documents, and with hard questions. Not “find this sentence”, but “compare clause 4 with clause 19” or “list every date mentioned”. That’s where weaknesses show up.
- Put the question and key material at the edges. Start or end, not the middle.
- Watch your bill and your latency. Long prompts are convenient during development and expensive in production.
- Ask vendors how they handle long inputs. Do they send everything, retrieve, or both? What happens when your data is bigger than the window?
And keep the Kalamang result in perspective. It’s genuinely impressive that a model learned to translate one direction of an almost undocumented language from one book in its prompt. It’s also one task, scored by one learner, with a real gap in the other direction. Both of those things are true at once.
Key ideas
- Attention compares every token with every other one, so the work grows with the square of the length. Double the text, four times the comparisons.
- The KV cache stores each token’s keys and values so the model doesn’t recompute them. It keeps full detail for every token in every layer, so it grows with length and can outweigh the model itself at long contexts.
- Long context comes from a toolbox: shared keys and values (GQA) shrink the cache, FlashAttention makes exact attention faster, sliding and sparse attention skip comparisons, and RoPE scaling teaches models positions they never saw in training.
- Needle-in-a-haystack tests only check finding one standout fact. Harder tests show many models fall short of their advertised length.
- Position matters. Models often use information at the start and end of a long prompt better than in the middle.
- Long context and RAG are a trade-off: long context can answer better, RAG is much cheaper, and combining them is often best.
Sources: Gemini Team, Google, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context”, technical report, Feb 2024 (arXiv:2403.05530, sections 4.2.1 and 4.2.2.1); Google, “Our next-generation model: Gemini 1.5”, 15 Feb 2024 (blog.google); Tanzer et al., “A Benchmark for Learning to Translate a New Language from One Grammar Book” (MTOB), 2023 (arXiv:2309.16575); Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023 (arXiv:2309.06180); Ainslie et al., “GQA”, 2023 (arXiv:2305.13245); Dao et al., “FlashAttention”, 2022 (arXiv:2205.14135); Jiang et al., “Mistral 7B”, 2023 (arXiv:2310.06825); Child et al., “Generating Long Sequences with Sparse Transformers”, 2019 (arXiv:1904.10509); Beltagy et al., “Longformer”, 2020 (arXiv:2004.05150); Su et al., “RoFormer”, 2021 (arXiv:2104.09864); Chen et al., “Extending Context Window of Large Language Models via Positional Interpolation”, 2023 (arXiv:2306.15595); Peng et al., “YaRN”, 2023 (arXiv:2309.00071); Hsieh et al., “RULER”, 2024 (arXiv:2404.06654); Liu et al., “Lost in the Middle”, 2023 (arXiv:2307.03172); Li et al., “Retrieval Augmented Generation or Long-Context LLMs?”, EMNLP 2024 (arXiv:2407.16833); Google, Gemini API context caching docs (ai.google.dev); Anthropic, prompt caching docs (docs.anthropic.com).
Related: Mixture of Experts: Why Giant Models Only Use Part of Their Brain, Shrinking AI: Quantization and Distillation Explained, and The Hidden Costs of AI Tools Nobody Shows in the Demo.
