# How AI Turns Words Into Numbers: Embeddings Explained > A token ID is just a label. An embedding is a learned list of numbers that puts each word on a map of meaning. Here's how it works, and where it shows up. - Author: Tai Pham Phu - Published: 27 Oct 2026 - Tags: How AI Works, LLM, Embeddings, Applied AI, Technology - URL: https://phamphutai.com/2026/10/27/embeddings-how-ai-turns-words-into-numbers/ In part 2 we cut our running sentence, "The cat sat on the mat because it was tired.", into tokens: small chunks of text, often whole words, sometimes pieces of words. Each token got a token ID, a whole number from the model's fixed vocabulary list. We also saw that a model can only look at so many tokens at once, its context window, and that this is why long chats start forgetting things and why you pay per token. So at this point the sentence is a short row of whole numbers. That's progress, but it's not understanding. This part is about the step that turns those numbers into something a model can actually reason with. --- ## A puzzle to start In 2016, a team from Boston University and Microsoft Research gave a computer a word puzzle. Man is to computer programmer as woman is to... what? The answer came back: homemaker. They tried another. Father is to doctor as mother is to: nurse. Nobody typed those stereotypes into the system. It wasn't a chatbot, and it had no rules about jobs or gender. It was a big table of numbers, learned automatically from Google News articles. So how does a table of numbers end up with something that looks like an opinion? The answer is the idea under every modern language model, including ChatGPT, Claude and Gemini. It's called an **embedding**. By the end of this post you'll know what one is, how it's learned, why the puzzle came out that way (it's less clear-cut than the headlines said), and how the same idea lets you search your own documents by meaning. --- ## A token ID is just a label Start with what we have after part 2: token IDs. A token ID works like a seat number in a stadium. Seat 9,059 and seat 9,060 are neighbours, but that tells you nothing about the people sitting in them. Token IDs are the same. In part 2, " cat" came out as token ID 9,059 with OpenAI's o200k_base tokenizer. "Kitten" gets some other number, and nothing about the two numbers says a kitten is a young cat. The numbers are just positions in a list. If a model only had labels, it would have to learn everything about "cat" separately from everything about "kitten", "cats" and "feline". Nothing it learned about one would help with the others. So the very first thing a language model does with a token ID is swap it for something richer: a long list of numbers that describes the token. That list is the **embedding**. Another word you'll see for "a list of numbers" is **vector**. In this series the two mean nearly the same thing: an embedding is the vector a model uses for a token. Mechanically, the swap is boring: the model keeps a giant table with one row per token, and a token ID just says which row to read. The interesting part is what those numbers are, and where they come from. --- ## A map of meaning The easiest way to picture an embedding is a map. Imagine every word gets a pin on a flat map, and each pin has two numbers: how far across and how far up. Now arrange the pins so that words with similar meanings sit close together. Animals in one corner. Things you put on a floor in another. Feelings somewhere else.

Illustrative only: a real model doesn't use two numbers per word but hundreds or thousands, and nobody hand-places the pins.

Now the map does something useful: **distance means similarity**. To find words like "cat", you look at what's nearby, and you find "dog" and "kitten". "Mat" sits next to "rug". "Kitten" and "happy" are far apart, because they don't have much to do with each other. Notice that our running sentence is spread across all three clusters. "Cat" is an animal, "mat" is a floor thing, and "tired" is a feeling. That will matter in part 4. ### Where the map analogy breaks Two numbers can't hold much. Is "tiger" closer to "cat" or to "zoo"? It depends on what you care about: what kind of thing it is, or where you'd find one. A flat map has to pick one answer. So real embeddings use many more directions. Add a third number, then a fourth, then hundreds more. Each direction can capture some shade of meaning: alive or not, big or small, formal or casual, singular or plural. One honest caveat: most of these directions don't line up with neat human labels like those. They're whatever turned out to be useful during training. But taken together they place each word very precisely. How many numbers are we talking about? Some real, published sizes: - The original word2vec vectors trained on Google News, the ones used in the 2016 puzzle, have **300** numbers per word, for about 3 million words and phrases. - The original transformer from 2017, the design behind today's chatbots, used **512** numbers per token. - The largest GPT-3 model (2020) used **12,288** numbers per token. - OpenAI's current embedding models, at the time of writing (October 2026), return **1,536** numbers (text-embedding-3-small) or **3,072** (text-embedding-3-large) by default. Nobody can picture 12,288 directions. You don't need to. The rule is the same as on the flat map: close together means similar. --- ## Similarity as closeness "Close together" needs one more word of explanation, because there are two ways to measure it. The obvious way is straight-line distance, like measuring between two pins with a ruler. That works. But the method you'll hear about most is **cosine similarity**, and it's easier than it sounds. Picture each embedding as an arrow from the centre of the map out to the word's pin. Cosine similarity ignores how long the arrows are and asks only: **do they point the same way?** It gives a score: - **1** means the arrows point in exactly the same direction: very similar. - **0** means they're at right angles: unrelated. - **−1** means they point in opposite directions. OpenAI's embeddings guide recommends cosine similarity and says the choice of distance function "typically doesn't matter much". The point to remember: **similarity is a score for how close two lists of numbers are**. --- ## How the map gets learned So who places the pins? Nobody, by hand. Embeddings are **learned during training**, not written by people. The idea behind it is older than computers doing it. In 1957 the linguist J. R. Firth wrote: "You shall know a word by the company it keeps." Try it yourself. "I fed the ___ some milk." "The ___ purred on my lap." "Cat" fits both. "Kitten" fits both. "Dog" fits the first one. "Rug" fits neither. Words that show up in the same kinds of sentences tend to mean similar things. In 2013, Tomas Mikolov and colleagues at Google turned that idea into a fast training method called **word2vec**. In plain terms, it works like this: 1. Give every word a random list of numbers. The map starts as a meaningless scatter. 2. Read a huge amount of text. For each word, play a guessing game: from this word, predict the words around it (or the other way round, from the surrounding words, predict the middle one). 3. Every time the guess is off, nudge the numbers a tiny bit so the guess would have been a little better. 4. Repeat billions of times. Words that keep the same company get nudged in the same directions, so they drift together on the map. Nobody tells the system that Paris and Rome are both capitals, or that "happy" and "glad" mean nearly the same thing. It works that out from the sentences they appear in. Modern chatbots don't use word2vec, but the principle carries over. Their embedding table starts random too and is learned as part of the model's main training, the next-token prediction game from part 1. Every time the model guesses the next token wrong, every number involved in that guess gets nudged, including the rows of the embedding table. The 2017 transformer paper says it plainly: it uses "learned embeddings" to turn tokens into vectors. Nobody writes a single one of those numbers by hand. --- ## Directions carry meaning: king − man + woman ≈ queen This is the result that made word embeddings famous. On the map, not only distances mean something. Directions do too. Start at "man" and walk to "king". You've walked in a direction that roughly means "royal". Now start at "woman" and take the same walk, same direction, same distance. Where do you land? Very close to "queen".

Illustrative. The walk from "man" to "king" and the walk from "woman" to "queen" point roughly the same way, so relationships become directions you can add and subtract.

Written as arithmetic: take the numbers for "king", subtract the numbers for "man", add the numbers for "woman", then look for the nearest word to the result. In the 2013 paper "Linguistic Regularities in Continuous Space Word Representations", Mikolov, Yih and Zweig (then at Microsoft Research) reported that king − man + woman gives a vector "very close to" queen. The word2vec paper from Google, published the same year, repeated the example and showed similar patterns for countries and capitals (France is to Paris as Germany is to Berlin) and for word forms (walking and walked, swimming and swam). The lesson is that the map doesn't only store *which* words are similar. It also stores *how* they're related, as directions. ### The caveats This result is real, but it's routinely oversold. Three things worth knowing: **It works only some of the time.** In the 2013 papers, the analogy tests were far from solved. The Microsoft Research paper reports answering "almost 40%" of its grammar analogy questions correctly. In the word2vec paper's comparison table, the best models scored in the 50s to around 60% on the meaning and grammar questions. Impressive for 2013, but "roughly half" is a fair summary, not "always". **There's a rule that quietly helps.** The standard analogy code is not allowed to return any of the words you put in. A 2020 paper by Nissim, van Noord and van der Goot tested what happens if you drop that rule. Very often the nearest word to king − man + woman is... king itself. Accuracy on the standard test fell sharply, from 0.71 to 0.21 with the usual method. "Queen" is close, but "king" is often closer, and the code just hides it. **That rule also shaped the famous bias result.** Back to the puzzle at the top of this post. The 2016 study by Bolukbasi and colleagues used the Google News word2vec vectors (300 numbers per word). Nissim and colleagues showed that with the input words allowed, the nearest answer to "man is to computer programmer as woman is to ___" is "computer programmer". The "homemaker" answer depends on the rule that forbids returning the input, plus choices about which words to search. That doesn't mean the bias is imaginary. Nissim and colleagues say outright that human biases are present in word embeddings. Bolukbasi's team showed something more solid than one analogy: gender appeared as an actual direction in the map, and job words like "receptionist" leaned along it. Because it was a direction, they could measure it, and reduce it for words that should be neutral while keeping it for words like "queen". They called this debiasing. The fair summary is this: **an embedding learns from human text, so it soaks up our knowledge and our patterns, including stereotypes.** Word puzzles are a flashy but unreliable way to show it. When an AI gives you a stereotyped answer, it's usually not an opinion. It's the shape of what it learned from. --- ## Order matters too: position gets added There's one problem with everything so far. An embedding table gives "cat" the same numbers wherever it appears. "The cat sat on the mat" and "the mat sat on the cat" contain exactly the same tokens, so they'd produce exactly the same set of embeddings. The model would have no idea which word came first. So transformers also feed in **position**. In the original 2017 transformer, each token's embedding has a second list of numbers added to it, one that encodes "I'm the 1st token", "I'm the 2nd token", and so on. The two lists are the same length, so they're simply added together, number by number. The paper tried both a fixed mathematical pattern and learned position numbers, and found they worked almost equally well. Many newer models handle position differently, mixing it in during the attention step rather than adding it at the start, but the purpose is the same: the model needs to know not just *which* words are present but *where*. We'll see where that happens in part 5. There's one more thing a single, fixed embedding can't do. "Bank" gets one row in the table, whether you mean a river bank or a bank account. And in our sentence, "it" gets the same starting numbers whether it refers to the cat or the mat. Fixing that, by letting each word's numbers be adjusted by the words around it, is the job of attention. That's part 4. --- ## The same idea, working for you: search by meaning Embeddings aren't just a step inside chatbots. Separate **embedding models**, like the OpenAI ones mentioned above, turn a whole sentence or paragraph into one list of numbers. Same map idea: texts that mean similar things land close together, even if they share no words. That makes **semantic search** possible, which means searching by meaning rather than by exact words. Type "can I bring my dog?" into a well-built help centre, and it can find the page titled "Pet policy", even though none of your words appear in it. Your question and that page land close together on the map. Here's how you'd search your own documents this way, in plain steps: 1. **Cut your documents into chunks.** A few paragraphs each: house rules, FAQ answers, policy sections, past email replies. 2. **Turn every chunk into an embedding** with an embedding model, once, and store the numbers next to the text. A spreadsheet is fine to start; a "vector database" is a database built to do this fast at scale. 3. **When a question comes in, embed the question too.** 4. **Find the closest chunks**, usually by cosine similarity, and take the top few. That alone gives you a search box that understands "dog" and "pet" are related. Add one more step, and you get **retrieval-augmented generation**, or **RAG**: 5. **Hand those chunks to a language model** along with the question, and ask it to answer *using only those chunks*. This is usually what's happening when a chatbot "reads your documents". It doesn't read them all every time. It retrieves the few pieces closest to your question on the map and answers from those. (The name comes from a 2020 paper by Lewis and colleagues.) Two practical warnings, because this is where people get burned: - **Similar isn't the same as correct.** The closest chunk is the one that *sounds* most like your question. "What's our refund policy for cancellations within 24 hours?" may pull up the general cancellation page instead of the one exception buried elsewhere. If the right chunk isn't retrieved, the model can't use it, and it may answer confidently anyway. - **Keep hard facts in code where you can.** Prices, dates and booking details are better looked up exactly than found by meaning. Use embeddings for the messy, wordy questions, and check what comes back. I go into that design rule in [Treat the LLM Like an Unreliable API](https://phamphutai.com/2026/10/14/treat-the-llm-like-an-unreliable-api/). --- ## Recap A token ID is a label, like a seat number. The model swaps each one for an **embedding**, a learned list of numbers that pins the token on a map of meaning with hundreds or thousands of directions. Close together means similar, and cosine similarity is the usual score for "how close". Directions can carry relationships, roughly and not always. The map is learned from the company words keep, so it inherits our knowledge and our biases. Position is added so the model knows word order. And the same idea lets you search your own documents by meaning. --- ## What to take away - **AI search finds meaning, not exact words.** Describe what you want naturally. You don't need to guess the right keyword. - **"Closest" is not "true".** Whether it's a search result or a retrieved paragraph, a high similarity score means "sounds related", nothing more. - **When an answer sounds like a stereotype, push back or rephrase.** You're often seeing the shape of the text the model learned from, not a fact. Our sentence, "The cat sat on the mat because it was tired.", is now a row of vectors, one per token, each with its position. But "it" still has the same numbers it would have in any sentence. It doesn't yet know whether it means the cat or the mat. **Next:** How AI Knows What "It" Means: Attention Explained. *Sources: Mikolov, Chen, Corrado, Dean, ["Efficient Estimation of Word Representations in Vector Space"](https://arxiv.org/abs/1301.3781) (Google, 2013); Mikolov, Yih, Zweig, ["Linguistic Regularities in Continuous Space Word Representations"](https://aclanthology.org/N13-1090/) (NAACL 2013); Bolukbasi, Chang, Zou, Saligrama, Kalai, ["Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings"](https://arxiv.org/abs/1607.06520) (NeurIPS 2016); Nissim, van Noord, van der Goot, ["Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor"](https://aclanthology.org/2020.cl-2.7/) (Computational Linguistics, 2020); Vaswani et al., ["Attention Is All You Need"](https://arxiv.org/abs/1706.03762) (2017); Su et al., ["RoFormer: Enhanced Transformer with Rotary Position Embedding"](https://arxiv.org/abs/2104.09864) (2021), an example of position mixed in at the attention step; Brown et al., ["Language Models are Few-Shot Learners"](https://arxiv.org/abs/2005.14165) (2020), model size table; Lewis et al., ["Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"](https://arxiv.org/abs/2005.11401) (2020); [OpenAI embeddings guide](https://developers.openai.com/api/docs/guides/embeddings), checked October 2026; J. R. Firth, "A Synopsis of Linguistic Theory, 1930–1955", in Studies in Linguistic Analysis (1957). The maps in this post are illustrations, not real embedding positions.* *Related: [How ChatGPT Actually Works, Explained Simply](https://phamphutai.com/2026/10/25/how-chatgpt-actually-works/), [What's a Token, and Why AI Has Limits](https://phamphutai.com/2026/10/26/what-is-a-token/), and [Treat the LLM Like an Unreliable API](https://phamphutai.com/2026/10/14/treat-the-llm-like-an-unreliable-api/).*