# How AI Sees: Vision Transformers and CLIP
> Why a paper note saying "iPod" fooled an AI looking at an apple, and how vision transformers, CLIP and chat models really turn pictures into tokens.
- Author: Tai Pham Phu
- Published: 20 Oct 2026
- Tags: How AI Works, LLM, Applied AI, Computer Vision, Multimodal AI
- URL: https://phamphutai.com/2026/10/20/how-ai-sees-vision-transformers-and-clip/
In March 2021, a team at OpenAI showed an AI a photo of an apple.
The AI was CLIP, a model that matches images with text. Asked what the photo showed, it was confident: **Granny Smith, 85.6%**. Correct, right down to the variety.
Then the researchers took a small piece of paper, wrote **"iPod"** on it by hand, and stuck it on the apple. Same apple, same table, same light. This time CLIP said **iPod, 99.7%**. Granny Smith dropped to about 0.1%.
They tried other words too. A note saying "library" turned the apple into a library, at 90.5%. They called these **typographic attacks**: fooling an AI with nothing more than a pen. In their write-up on Distill, they pointed out that once you know the trick, anyone can use it, "including six year olds".
It sounds like a silly bug. It's actually a window into how these models see. CLIP didn't fail because it was bad at seeing apples. It failed because it had learned something impressive, how to read, from the way it was trained. To understand why, we need to look at how a machine that was built for words learned to look at pictures.
If tokens, embeddings and attention are new to you, start with the basics first. This post assumes you know roughly how a transformer reads text.
---
## The problem: a picture isn't a sentence
A language model reads tokens. A sentence becomes a sequence of token IDs, each one becomes an embedding (a list of numbers that captures its meaning), and attention lets every token look at every other token.
An image is different. It's a grid of pixels, each with three numbers for red, green and blue. A modest 224 × 224 photo is about 150,000 numbers, with no obvious "words" in it.
For about a decade, the answer to images was a different kind of network, the **convolutional neural network** (CNN). A CNN slides small filters across the image, looking for edges, then textures, then shapes, building up from local details. AlexNet, the 2012 network from the [previous post](https://phamphutai.com/2026/10/20/why-ai-runs-on-gpus/), was a CNN. CNNs come with built-in assumptions about images: nearby pixels matter to each other, and a cat in the corner is still a cat.
Then transformers took over language, and researchers asked the obvious question. Could the same architecture just read images too?
---
## Vision Transformers: an image is worth 16×16 words
In October 2020, a Google team led by Alexey Dosovitskiy published a paper with a title that explains the idea: **"An Image is Worth 16x16 Words"**. Their model was the **Vision Transformer**, or ViT.
The trick is almost embarrassingly simple:
1. **Cut the image into squares.** Each square, called a patch, is 16 × 16 pixels. A 224 × 224 image becomes a 14 × 14 grid, so **196 patches**.
2. **Flatten each patch into a list of numbers.** 16 × 16 pixels × 3 colours = 768 numbers.
3. **Turn each patch into an embedding** with a learned transformation, exactly like turning a token ID into an embedding.
4. **Add position information**, so the model knows which patch came from where.
5. **Feed the sequence into a standard transformer.** Now patches attend to patches, the way words attend to words.
That's it. To the transformer, an image is a 196-word sentence where each "word" is a little square of picture.
### The catch: it needs a lot of data
ViT has far fewer built-in assumptions about images than a CNN. The paper says this directly: the only image-specific choices are cutting the picture into patches and adjusting positions for different resolutions. Everything else, including which patches relate to which, the model has to learn.
That cuts both ways. Trained on ImageNet alone (about 1.3 million images), ViT came in a few percentage points **below** comparable CNNs. Trained first on much larger datasets of 14 million to 300 million images, it caught up and passed them. The best version reached 88.55% accuracy on ImageNet. In the authors' words, large-scale training beat built-in assumptions.
This is a pattern we've seen across AI: a simpler, more general design that loses on small data and wins on big data. It's the same story as language models.
---
## CLIP: learning from captions instead of labels
ViT still learned the old way: from images with a fixed set of labels, like "golden retriever" or "fire truck". Labelling is slow and expensive, and you're stuck with the categories someone chose in advance.
OpenAI's **CLIP** (Contrastive Language-Image Pre-training), published in February 2021 by Alec Radford and colleagues, took a different approach. The internet is full of images that already have text next to them: captions, alt text, titles. Why not learn from that?
They collected **400 million image-text pairs** from the internet and trained two networks side by side:
- An **image encoder** (they tried both CNNs and Vision Transformers) that turns a picture into a vector.
- A **text encoder** (a transformer) that turns a caption into a vector of the same size.
The goal: make the vector for each image land close to the vector for its own caption, and far from everyone else's.
### How contrastive training works
Take a batch of image-caption pairs. CLIP used batches of **32,768** at a time. Encode every image and every caption. Now compare every image with every caption, which gives a big grid of similarity scores.
- The **matching pairs**, on the diagonal of the grid, should score high.
- **Every other combination** should score low. With N pairs in a batch, that's N correct matches and N² − N wrong ones.
Training nudges both encoders to push the right pairs together and the wrong pairs apart. That's what "contrastive" means: the model learns by contrast, by telling the right answer apart from thousands of wrong ones.
CLIP compares every image in a batch with every caption, and learns to score the true pairs high and every other pairing low.
The paper also explains why they chose this. Their first attempt tried to make the model predict the exact words of each caption. That was slow, because there are many valid ways to describe a picture. Simply asking "which caption goes with this image?" was several times more efficient.
It was still expensive. The paper reports that the largest CNN version took 18 days on 592 NVIDIA V100 GPUs, and the largest Vision Transformer version took 12 days on 256 of them.
### The payoff: zero-shot classification
Once text and images live in the same space, you can classify images without ever training a classifier. This is called **zero-shot** classification: doing a task with zero examples of it.
Say you want to sort photos into dogs, cats and birds. Write one sentence per category, like **"A photo of a dog."**, encode each sentence with the text encoder, encode the photo with the image encoder, and pick the sentence whose vector is closest. That's the whole classifier.
The results surprised people. On ImageNet, zero-shot CLIP reached **76.2%** accuracy, matching the original ResNet-50, a classic CNN that was trained directly on ImageNet's 1.28 million labelled examples. CLIP used none of them.
Even the wording matters. The researchers found that "A photo of a {label}." worked better than the bare label, because captions on the internet are usually sentences, not single words. That small template alone added 1.3 percentage points on ImageNet. It's prompt engineering, for images, in 2021.
---
## Why the note saying "iPod" worked
Now back to the apple.
Think about what's in those 400 million internet captions. A photo of a shop sign, captioned with the shop's name. A book cover, captioned with the title. A T-shirt with a slogan, captioned with the slogan. **The text in the picture often is the caption.** So the easiest way to match many images to their captions is to learn to read.
CLIP learned to read. When the OpenAI team looked inside the model, they found what they called **multimodal neurons**: single units that respond to a concept whether it appears as a photo, a drawing or written text. One, which they nicknamed the Spider-Man neuron, responded to a photo of a spider, to the written word "spider", and to Spider-Man himself, in costume or drawn. A "finance" neuron responded to piggy banks and to the string "$$$". By making that neuron fire, they could make the model call a dog a piggy bank.
So when CLIP saw an apple with "iPod" written on it, two signals competed: the round green fruit, and a very strong, very clean "iPod" signal from the text. The text won.
The researchers also built an automated version of the attack on ImageNet images. Their most effective text snippets fooled the model **97%** of the time while changing only about 7% of the image's pixels.
The model wasn't broken. It was doing exactly what it was trained to do: find the caption that best matches the picture. A picture of an apple with "iPod" written on it is, arguably, best described by "iPod". The problem is that this isn't what we meant.
---
## How chat models see your images
CLIP itself can't chat. It only scores how well an image matches a piece of text. So how does a chatbot answer questions about a photo you upload?
Here's the common recipe, as described in open research. The clearest public example is **LLaVA** (Liu et al., 2023):
1. **An image encoder turns the picture into patch vectors.** LLaVA used CLIP's Vision Transformer, the version with 14 × 14 pixel patches. Out come one vector per patch.
2. **A small "projector" translates those vectors into the language model's embedding space.** In LLaVA it was a single learned matrix. Its job is to make image vectors look like the kind of input the language model already understands.
3. **The translated vectors go into the language model as "visual tokens"**, right next to your text tokens. From there, it's a normal transformer. When you ask "what colour is the car?", the text tokens can attend to the image tokens just as they attend to earlier words.
Training happened in two stages. First, LLaVA kept both the image encoder and the language model frozen, and trained only the projector on image-caption pairs. The paper compares this stage to building a tokenizer for images that the frozen language model can understand. Second, they fine-tuned the projector and the language model together on conversations about images, keeping the image encoder frozen.
In the common open design, an image encoder and a small projector turn an image into "visual tokens" that sit in the same sequence as your words.
This isn't the only design, and it's worth being precise about what we know:
- **Other open designs exist.** DeepMind's Flamingo (2022) let the language model look at image features through extra cross-attention layers, instead of putting image tokens in the main sequence.
- **Some models are trained on images from the start.** Google's Gemini report describes its models as natively multimodal: trained jointly on text, images, audio and video from the beginning, with visual encoding inspired by Flamingo and other Google work.
- **Most commercial labs don't publish their exact architecture.** So when someone tells you exactly how a particular closed chatbot sees, be careful. What we can say is that the patches-into-tokens idea is widespread.
One provider does document the cost side. Anthropic's docs (at the time of writing, October 2026) say Claude views images in **28 × 28 pixel patches**, each counted as a visual token. A 1000 × 1000 pixel image costs **1,296 tokens**, and large images are scaled down to a per-model limit. That's the patch idea from ViT, showing up on your bill.
---
## Limits: what these models still can't see
Vision-language models are remarkable at describing scenes, reading documents and explaining charts. They are also surprisingly bad at some things a child does easily.
**Counting.** The CLIP paper itself lists counting the number of objects in an image as a weakness. Anthropic's own documentation says Claude's counts are approximate and may be off, especially when there are many small objects.
**Precise spatial relationships.** In a 2024 study titled "Vision language models are blind", researchers built seven very simple tests: do two circles overlap, how many times do two lines cross, which letter in a word is circled, how many rings are in an Olympic-style logo. Four leading models of the time (including GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet) averaged **58%** accuracy. The best, Claude 3.5 Sonnet, got 78%. A person would score close to 100%. The models did fine when the shapes were spread out, and struggled when they overlapped or sat close together.
The interesting finding: when the researchers tested the image encoders directly, the information needed to answer was there. The language model just failed to get it out. Seeing and describing are different skills.
**Blind spots inherited from CLIP.** Another 2024 study, "Eyes Wide Shut?" from NYU (with Yann LeCun among the authors), collected pairs of images that CLIP treats as nearly the same even though they look clearly different, and turned them into simple questions, such as which way a dog is facing. Chat models, including GPT-4V, struggled with these questions and often gave confident, made-up explanations, and the patterns that tripped up CLIP lined up with the ones that tripped up the chat models. If the encoder can't tell two images apart, the language model on top has no chance.
**Reading too much.** The typographic attack never fully went away as an idea. A model that reads text in images will also read text you didn't mean as an instruction or label. That makes text inside images one more input you shouldn't blindly trust.
**Out-of-distribution images.** CLIP scored only 88% on MNIST, simple handwritten digits, and a basic textbook method on raw pixels beat it. The paper explains why: there were almost no images like that in its training data. Models see well what they've seen a lot of.
---
## What it means in practice
If you're building with vision models or buying a product that uses them:
- **Treat images as tokens with a price.** Resolution drives cost. Resize to what the task needs: a receipt needs readable text, a "is there a person in this photo" check doesn't need 4K.
- **Crop for small details.** Text and small objects can be lost when a large image is scaled down. Cropping the relevant area often helps more than a cleverer prompt.
- **Don't trust counts or exact positions without checking.** If your workflow depends on "how many items are on this shelf" or "is the box left of the line", test it on your own images and build in a human check.
- **Watch for text in images.** A model may give more weight to a label, sign or sticker than to what the object actually is. If users can upload images, assume someone will try a typographic trick, and design so a wrong reading can't trigger anything important on its own. I wrote more about that mindset in [Treat the LLM Like an Unreliable API](https://phamphutai.com/2026/10/14/treat-the-llm-like-an-unreliable-api/).
- **Test on your kind of images.** Models are strongest on images that look like the internet. Medical scans, technical drawings, unusual layouts and handwriting can be much weaker.
---
## Key ideas
- **Vision Transformers** cut an image into patches (for example 16 × 16 pixels) and treat each patch like a word token.
- ViTs have fewer built-in assumptions than CNNs, so they **need more data**, and win once they get it.
- **CLIP** learns from 400 million image-caption pairs by matching each image to its own caption and away from all the others: **contrastive training**.
- Shared image-text space enables **zero-shot classification**: describe the categories in words and pick the closest. CLIP matched a classic ImageNet model without its labels.
- CLIP **learned to read**, which is why a handwritten "iPod" note beat a real apple: a **typographic attack**.
- Many chat models see through an **image encoder plus a projector** that turns patches into **visual tokens** in the language model's input; closed models' exact designs are mostly unpublished.
- Weak spots remain: **counting, precise spatial relations** and images unlike the training data.
---
*Sources: Goh et al., ["Multimodal Neurons in Artificial Neural Networks"](https://distill.pub/2021/multimodal-neurons/) (Distill, 4 March 2021; apple figures from the article's typographic attack data for the CLIP RN50-4x model, zero-shot) and OpenAI's [companion blog post](https://openai.com/index/multimodal-neurons/); Dosovitskiy et al., ["An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale"](https://arxiv.org/abs/2010.11929) (2020, ICLR 2021); Radford et al., ["Learning Transferable Visual Models From Natural Language Supervision"](https://arxiv.org/abs/2103.00020) (CLIP, 2021); Liu et al., ["Visual Instruction Tuning"](https://arxiv.org/abs/2304.08485) (LLaVA, 2023); Alayrac et al., ["Flamingo: a Visual Language Model for Few-Shot Learning"](https://arxiv.org/abs/2204.14198) (2022); Gemini Team, Google, ["Gemini: A Family of Highly Capable Multimodal Models"](https://arxiv.org/abs/2312.11805) (2023); Anthropic, [Vision documentation](https://platform.claude.com/docs/en/build-with-claude/vision), checked October 2026; Rahmanzadehgervi et al., ["Vision language models are blind"](https://arxiv.org/abs/2407.06581) (2024); Tong et al., ["Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs"](https://arxiv.org/abs/2401.06209) (2024).*
*Related: [Why AI Runs on GPUs](https://phamphutai.com/2026/10/20/why-ai-runs-on-gpus/), [How Image Generators Work: Diffusion Explained](https://phamphutai.com/2026/10/20/how-image-generators-work-diffusion/), and [Treat the LLM Like an Unreliable API](https://phamphutai.com/2026/10/14/treat-the-llm-like-an-unreliable-api/).*