How to Read an ML Paper in One Hour

Open “Attention Is All You Need”, the 2017 paper that introduced the Transformer. The abstract says the big model reached 41.8 BLEU on English-to-French translation. Turn to section 6.1 of the current arXiv version (v7) and the same result is written as 41.0. Table 2, on the facing page, says 41.8.

It’s a small slip that doesn’t change the paper’s story. But you only catch it by reading the paper itself, not a summary of a summary.

This post is a method for doing that without losing a whole afternoon. It’s built on a short classic, S. Keshav’s “How to Read a Paper” (ACM SIGCOMM Computer Communication Review, 2007), adapted to machine learning papers and squeezed into one hour. At the end, I walk through the Transformer paper with it.

Why a developer should read papers at all

Because headlines compress, and compression loses the conditions. A paper says “on this dataset, with this much compute, against these baselines, our method scored X.” A headline says “new model beats GPT.” The conditions are exactly the part you need when deciding whether an idea will work in your product.

• You can check a claim before you build on it. Many “state of the art” results are real but narrow. The paper tells you how narrow.
• You learn what actually moved the number. A good paper has ablations (more on those below) that show which part of the method did the work. That part is often small and copyable.
• You stop being surprised by benchmarks. Benchmarks and Goodhart’s Law covers how scores break through saturation, contamination and gaming. Papers are where you can see it happen.

You don’t need a PhD for this. You need a method, a timer, and permission to not understand everything on the first read.

The anatomy of an ML paper

Most ML papers share one shape. Knowing it lets you jump to the part you need.

Two terms worth defining once:

• A baseline is the method you compare against. A new method only looks good relative to its baselines, so weak baselines make anything look good.
• An ablation is an experiment where you remove or swap one component and see how much the result drops. If removing the new idea barely changes the score, the new idea isn’t what produced the score.

Not every paper has a limitations section. Older ones often don’t. Many venues now ask for one: the NeurIPS paper checklist asks authors to discuss limitations, and ACL conferences require a limitations section.

The three-pass method, fitted to one hour

Keshav’s key idea is simple: don’t read a paper once, front to back. Read it in up to three passes, each with a different goal. Here’s his version, from the paper itself:

• First pass, about five to ten minutes. Read the title, abstract and introduction carefully. Read the section and sub-section headings and ignore everything else. Read the conclusions. Glance over the references.
• Second pass, up to an hour. Read with more care but skip details like proofs. Look carefully at figures and graphs. Are the axes labelled? Are there error bars? Mark references you haven’t read.
• Third pass, about four or five hours for beginners and about an hour for an experienced reader. Try to “virtually re-implement” the paper: make the same assumptions as the authors and re-create the work in your head, then compare. This is where hidden assumptions show up.

At the end of the first pass, Keshav says you should be able to answer the five Cs: Category (what kind of paper is this?), Context (what other papers is it related to?), Correctness (do the assumptions look valid?), Contributions (what are the main ones?) and Clarity (is it well written?).

For ML papers, I’d fit it into one hour like this. The third pass doesn’t fit in an hour, and that’s fine. Most papers don’t deserve one.

Pass 1: ten minutes, “should I read this?”

Set a timer. Read only:

1. Title and abstract. Write down the main claim in one sentence, in your own words.
2. The contributions list at the end of the introduction. Most ML papers have one. It’s the paper’s own table of contents for its claims.
3. Section headings. They tell you where the evidence lives.
4. Every figure and its caption, without reading the surrounding text. In ML papers, Figure 1 is often the whole method in one picture.
5. The main results table. Find the row with the new method and the rows it beats.
6. The conclusion.

Then answer the five Cs and decide: stop, come back later, or do pass 2. Stopping is a fine result.

Pass 2: forty minutes, “what exactly is claimed, and what’s the evidence?”

Now read the method and experiments sections properly, still skipping proofs and long derivations.

• Method: understand it at the level of the main figure plus one equation. If you could explain it on a whiteboard in two minutes, that’s enough for now.
• Experiments: read the setup paragraph carefully. Which datasets? Which baselines? Which metric? How was the final number chosen?
• Ablations: find the table that removes the new idea. This is the most important table in most papers.
• Limitations: read them, or notice there are none.
• References: mark two or three you’d read next.

Use the last ten minutes for notes (template below).

Pass 3: later, only if it earned it

Keshav’s “virtually re-implement” pass. For ML you can often run the code instead: re-derive the key step, read the appendix, and try the method on a slice of your own data. Save it for papers you’re about to build on.

What to check in the experiments

Pass 2 is where you decide whether to believe the paper. Here’s the checklist I’d use, with the reason behind each item.

Are the baselines strong and tuned?

A new method is tuned by people who want it to win. Baselines often aren’t. A sharp example: in 2017, Melis, Dyer and Blunsom (“On the State of the Art of Evaluation in Neural Language Models”, ICLR 2018) re-tuned standard LSTMs with large-scale automatic hyperparameter search and found they outperformed more recent architectures that had been reported as better.

Ask: were baselines re-run with the same tuning budget, or were their numbers copied from older papers with different setups? Copying is normal and often unavoidable, but it means the comparison isn’t apples to apples.

What was the compute budget?

If the new method used far more compute than the baseline, a better score might just be a bigger budget. Look for training time, hardware, parameter count and steps, for both sides.

How many runs, and how much variance?

ML training is noisy. Different random seeds (the number that sets initial weights and data order) give different results. In “Deep Reinforcement Learning that Matters” (Henderson et al., AAAI 2018), the authors ran the same algorithm with the same hyperparameters ten times, changing only the seed, and split the runs into two groups of five. The two groups’ averages were significantly different by a t-test. Same code, same settings, different luck.

So ask: is each number a single run? Are there error bars or standard deviations? Is the improvement bigger than the spread?

Could the test set have leaked?

Contamination means test data showed up in training data, so the model saw the answers. Public benchmarks live on the web, and large models train on web crawls. Look for an overlap check. More in the Goodhart post.

Are the examples cherry-picked?

Qualitative examples (sample outputs, images, attention maps) are picked by the authors. They show what the method can do, not what it usually does. Unless the paper says “randomly selected” or “uncurated”, assume the best ones were chosen.

Is the benchmark saturated?

When every method scores near the top, a small gain may be noise or label errors. Check how much room was left above the best baseline.

Do the ablations isolate the claim?

The paper’s claim is usually “component X causes improvement Y.” The ablation should remove X and nothing else. If the ablation table only varies sizes and learning rates but never removes X, the claim is untested.

Authors’ claim or independent replication?

The authors’ claim is one data point. Another team reproducing it with their own code is much stronger. Check whether later papers use the method as a baseline and get similar numbers.

Reading figures and tables

Most of the evidence in an ML paper is in tables and plots. A few habits help:

• Read the caption, axis labels and legend first. Only then the data.
• Check which split each number comes from. Development (validation) sets are for tuning. Test sets are for the final number. Results from different splits aren’t comparable.
• Check the y-axis start. A plot that starts at 80% makes a 1-point gap look huge.
• Find the cost columns. Parameters, FLOPs (floating-point operations, a rough measure of compute) and training time. Are they measured or estimated? The paper should say.
• Look for empty cells. A missing combination is a question worth asking.

Surviving the notation

You don’t need to follow every line of math. You need the one equation the paper is about.

• Build a symbol table. On your notes page, write each symbol and its meaning the first time it appears. Papers define symbols once and then use them for ten pages.
• Track shapes. For anything bold or capitalised, write its shape: a vector of length d, a matrix of n by d. Most ML equations become clear once you know the shapes.
• Read the equation as a sentence. Translate it into words: “take the dot product of each query with each key, scale it, turn the scores into weights, and average the values with those weights.”
• Plug in tiny numbers. Two tokens, two dimensions. Work it through on paper or in a Python shell.

If the basics (vectors, dot products, softmax, gradients) are the blocker, The Math Behind AI covers them from the code side.

Tools for finding and reading

All of these exist at the time of writing (November 2026):

• arXiv. Where most ML papers appear first, as preprints. Note the version number (v1, v2…): papers change between versions. Many recent papers also have an HTML view.
• Semantic Scholar. A free paper search engine from the Allen Institute for AI, with citation lists and a one-sentence auto-generated “TLDR” for many papers. Good for the “Context” C.
• Google Scholar. “Cited by” finds follow-up work and replications.
• OpenReview. Several conferences, including ICLR, publish their reviews here. Reading the reviewers’ objections is like getting a free expert second pass.
• Hugging Face Papers. Daily trending papers, often linked to models, datasets and code. The older Papers with Code site was shut down in July 2025 and now redirects to Hugging Face.
• The paper’s own code link. Usually in the abstract, a footnote or the conclusion. No code makes reproduction much harder.

A one-page notes template

Fill this in during the last ten minutes. One page, no more. If you can’t fill a field, that’s information too.

PAPER:        title, authors, year, arXiv ID + version
CATEGORY:     new method / new dataset / analysis / survey / system
ONE-LINE CLAIM (my words):

CONTRIBUTIONS (theirs, max 3):
  1.
  2.
  3.

KEY IDEA (whiteboard version, 2-3 sentences):

KEY EQUATION + symbol table:

EVIDENCE
  Datasets / splits:
  Baselines (re-run or copied?):
  Compute (theirs vs baselines):
  Runs / seeds / error bars:
  Ablation that removes the new idea:
  Contamination check:

RED FLAGS:
OPEN QUESTIONS:
CODE / REPLICATIONS:
READ NEXT (2-3 refs):
USE FOR ME: what would I try, on what data, measured how?

The last line matters most. A paper you can’t connect to an experiment is trivia.

Worked walkthrough: “Attention Is All You Need”

Let’s run the method on Vaswani et al., 2017 (arXiv 1706.03762). It’s 15 pages in the arXiv version, including references and attention visualizations at the end. Everything below is from the current arXiv version (v7).

Pass 1 (ten minutes)

• Title and abstract. One-line claim: a translation model “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”, which is better and faster to train than previous models. The abstract claims 28.4 BLEU on WMT 2014 English-to-German and a single-model state of the art on English-to-French.
• Headings. Introduction, Background, Model Architecture, Why Self-Attention, Training, Results, Conclusion. Notice what’s not there: no separate limitations section. That was normal in 2017.
• Figures. Figure 1 is the whole architecture: an encoder stack and a decoder stack. Figure 2 shows scaled dot-product attention and multi-head attention. Two figures, and you’ve seen the method.
• Main results. Table 2 lists BLEU scores and training cost for the Transformer next to earlier models.
• Conclusion. It repeats the translation results, says the authors plan to apply attention to images, audio and video, and links the code (the tensor2tensor repository on GitHub).

Five Cs: a new architecture; builds on recurrent and convolutional models with attention; nothing obviously off yet; contributions are the architecture plus the training-cost claim; short and clear. Decision: pass 2.

Pass 2 (forty minutes)

Method. Read section 3.2.1 and equation 1:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Symbol table: Q holds the queries, K the keys, V the values, and d_k is the key dimension. In words: score each query against every key, scale the scores down, turn them into weights with softmax, and take a weighted average of the values.

Now notice a sentence right after the equation: “We suspect” that for large d_k the dot products grow large and push softmax into regions with tiny gradients, so they scale by 1/√d_k. That marks a hypothesis with a supporting argument (a footnote about variance), not a measured result. Good readers notice hedges.

Experiments. Section 5 gives the setup: one machine with 8 NVIDIA P100 GPUs. The base model trained for 100,000 steps, or 12 hours. The big model trained for 300,000 steps, or 3.5 days. Section 6.1 adds details that matter for comparison: the final models average the last 5 checkpoints (base) or the last 20 (big), and beam size and length penalty were chosen on the development set.

Baselines and compute. Table 2 compares against ByteNet, GNMT, ConvS2S, a mixture-of-experts model and some ensembles. These numbers are cited from the earlier papers, not re-run. That’s common, but it means each row comes from a different setup. The training-cost column is in FLOPs. The paper says it estimates them by multiplying training time, number of GPUs and an assumed sustained throughput per GPU (9.5 TFLOPS for a P100, per a footnote). So it’s an estimate, not a measurement.

The 41.8 vs 41.0 mismatch. The abstract and Table 2 say 41.8 for English-to-French, and section 6.1 says 41.0. Only a reader who cross-checks prose against tables catches it. Quote numbers from tables.

Variance. The tables give one number per configuration, with no error bars or spread across seeds, which was typical in 2017.

Ablations. Table 3 is the ablation table, measured on the development set (newstest2013) without checkpoint averaging, so its numbers (25.8 BLEU for the base model) can’t be compared directly with Table 2 (27.3 for the base model on newstest2014). Then read the rows:

• Rows (A) vary the number of attention heads at constant compute. A single head is 0.9 BLEU worse than the best setting, and too many heads also hurts.
• Rows (B) show that reducing the key size d_k hurts quality.
• Rows (C) and (D): bigger models do better, and dropout helps.
• Row (E) swaps the sinusoidal position encoding for learned position embeddings and gets nearly identical results.

Row (E) is useful: the sinusoidal encoding, which takes real space in the method section, isn’t what makes the model work. And the 0.9-point gap in rows (A) is one run each. Bigger than seed noise? The paper can’t say.

Also notice what Table 3 can’t test: you can’t “remove attention” from a model that is only attention. The evidence for the main claim is the comparison with other architectures in Table 2, which brings back the copied-baselines caveat.

Generalisation. Section 6.3 tests English constituency parsing (turning sentences into grammar trees) with little task-specific tuning, and honestly reports one earlier model it doesn’t beat, the Recurrent Neural Network Grammar.

Figures 3 to 5. The attention visualizations at the end show heads that seem to track long-distance dependencies and pronoun references. The captions hedge (“apparently”, “seems related to”). These are chosen examples, so treat them as illustrations of what heads can learn.

Replication. Easy for this paper. The authors released code, and independent re-implementations exist, such as Harvard NLP’s “The Annotated Transformer” (Sasha Rush, 2018, updated in 2022), which rebuilds the model line by line beside the paper’s text. And modern large language models are built on the architecture, which is replication at a scale few papers get.

Would it earn pass 3?

For most developers, yes, but in a specific form: implement scaled dot-product attention yourself on two tokens with NumPy, then step through The Annotated Transformer. Attention: How AI Knows What It Means and Inside a Transformer cover the ideas if you’d rather start from the picture.

When a paper is worth your time

You’ll never read everything. Here’s a quick filter before pass 1:

• It answers a question you actually have, not “what’s new this week?”
• It has code, or enough detail to rebuild. If not, you can learn the idea but not test it.
• Others are building on it. Check “Cited by” and whether it shows up as a baseline in later work.
• The claim is about mechanism, not just a score. “X works because Y, shown by ablation Z” teaches you something. “We scored 0.4 higher” often doesn’t transfer.
• The setting is close to yours. A pretraining trick may not help a model you call through an API.

In an unfamiliar area, a recent survey paper is often the best first read. Keshav agrees.

Turning reading into experiments

Reading is half the value. The other half is the experiment the paper suggests for your system. The pattern I’d use:

1. Pick one claim from your notes, the one most relevant to what you’re building.
2. Shrink it. What’s the smallest version you can test in a day? One component, one dataset you own, one metric you trust.
3. Write down the prediction first. “If the paper is right, my retrieval hit rate should go up on these 50 queries.” This keeps you honest.
4. Run the baseline yourself. Same budget, same data, same metric. Don’t compare your new thing to a number from the paper.
5. Run more than once if anything is random, and look at the spread before you celebrate.

For a full example of this on a real project, see What the Research Says, where I read four research fields against my own results and turned them into a list of experiments to run next. The same discipline applies to everyday product work, like evaluating AI features before you ship them.

Key ideas

• Read in passes, not front to back. Ten minutes to decide, forty to understand the claim and evidence, a separate session only if it earns it.
• The contributions list and the ablation table are the two most useful parts of most ML papers.
• Check baselines, compute, seeds and contamination before you believe a gain. Weakly tuned baselines and single runs make small improvements unreliable.
• Quote numbers from tables, not prose, and note the arXiv version you read.
• Hedge words like “we suspect” and “apparently” mark hypotheses. Respect them.
• Fill in a one-page note every time, ending with “what would I try, on what data?”
• A paper becomes useful when it becomes an experiment on your own system, with your own baseline.

Sources: S. Keshav, “How to Read a Paper”, ACM SIGCOMM Computer Communication Review 37(3), 2007; A. Vaswani et al., “Attention Is All You Need”, NeurIPS 2017 (arXiv v7, August 2023); G. Melis, C. Dyer, P. Blunsom, “On the State of the Art of Evaluation in Neural Language Models”, ICLR 2018; P. Henderson et al., “Deep Reinforcement Learning that Matters”, AAAI 2018; Harvard NLP, “The Annotated Transformer”; Semantic Scholar TLDR.

Related: Benchmarks and Goodhart’s Law, The Math Behind AI, and What the Research Says.