# Shrinking AI: Quantization and Distillation Explained > How big AI models get small enough for a laptop or phone: quantization, distillation, pruning, what each one costs, and the licence questions. - Author: Tai Pham Phu - Published: 20 Oct 2026 - Tags: How AI Works, LLM, Applied AI, Quantization, Distillation - URL: https://phamphutai.com/2026/10/20/quantization-and-distillation/ In March 2023, a small team at Stanford published a blog post about a model called **Alpaca**. They started with LLaMA 7B, a 7-billion-parameter model Meta had released to researchers. On its own, it was a "base" model: good at continuing text, not good at following instructions. To teach it, they needed examples of instructions and good answers. Writing tens of thousands by hand would take months. So they asked a bigger model to write them. They began with 175 instruction-and-answer pairs written by people, then had OpenAI's text-davinci-003 generate more in the same style. The result was 52,000 examples. Generating them through OpenAI's API cost **less than $500**. Fine-tuning LLaMA 7B on them took **3 hours on 8 high-end GPUs**, which the team estimated at **under $100** on most cloud providers. Under $600 in total. And in a small blind test, where five of the student authors compared answers side by side, Alpaca won 90 comparisons and text-davinci-003 won 89. That made headlines. A model you could run on modest hardware, trained for the price of a nice dinner for a team, roughly matching a big commercial model. The full story is more careful than the headlines. The team themselves wrote that the evaluation was limited in scale and diversity, that Alpaca hallucinated often (one example confidently gives the wrong capital of Tanzania), and that it was **for academic research only**. They gave three reasons it couldn't be used commercially: LLaMA's licence was non-commercial, OpenAI's terms prohibited using its outputs to build competing models, and they hadn't built proper safety measures. Their public demo has since been taken down, which they put down to hosting costs and weak content filters. But the core idea stuck, and it's now everywhere: **a big model can teach a small one**. Together with **quantization** (storing the model's numbers with less precision), this is how capable AI ends up on laptops and phones. This post explains both, plus pruning, and what you give up with each. If tokens or fine-tuning are new to you, start with the basics first. --- ## The problem: models are mostly memory A language model is, physically, a giant list of numbers called **parameters** (or weights). A "7B" model has about 7 billion of them. To run it, every one has to sit in memory, and for every word it writes, the chip has to read through them. How much memory depends on how precisely each number is stored: - **32-bit** ("full precision"): 4 bytes per number. A 7B model needs about **28 GB**. - **16-bit** (the usual format for running models): 2 bytes. About **14 GB**. - **8-bit**: 1 byte. About **7 GB**. - **4-bit**: half a byte. About **3.5 GB**. That's simple arithmetic: parameters times bytes per parameter. Real files are a bit larger, and you also need room for the KV cache (the model's memory of the conversation, covered in [the previous part](https://phamphutai.com/2026/10/20/long-context-and-the-kv-cache/)).

Same model, same number of parameters: only the precision of each number changes.

A 70B model in 16-bit needs about 140 GB, more than any single consumer graphics card. In 4-bit, about 35 GB, which fits on a high-end workstation or a Mac with lots of memory. There's a second reason size matters: **speed**. When a model generates text, the chip spends much of its time just moving weights from memory to the processor. Fewer bytes to move means faster answers. Memory, not maths, is often the bottleneck. So there are two ways to shrink a model. Store each number with fewer bits (**quantization**). Or have fewer numbers (**distillation** into a smaller model, or **pruning**). --- ## Quantization: rounding, done carefully Quantization means storing each weight with less precision. The everyday version: instead of saying a price is $19.9873, you say $20. You've lost a little detail, but the number is shorter. For model weights, 16-bit numbers can represent tens of thousands of distinct values. A 4-bit number can only represent **16**. So quantizing to 4 bits means: for each small group of weights, pick 16 levels that cover their range, and snap every weight to the nearest level. You also store a scale for each group, so the model can roughly rebuild the original values when it runs. ### What gets lost Each weight gets a small rounding error. Spread over billions of weights, those errors add up, and the model gets slightly worse at predicting text. How much worse depends on three things: 1. **How few bits.** 8-bit is usually very close to the original. 4-bit is typically a good trade. Below that, quality drops faster. 2. **How big the model is.** Larger models tend to handle it better. The GPTQ paper (below) notes that larger models are generally easier to quantize. Smaller models have less room for error, so the same squeeze hurts them more. 3. **How smart the rounding is.** This is where the named methods come in. One reason naive rounding fails: **outliers**. A 2022 paper by Tim Dettmers and colleagues, LLM.int8(), found that large models develop a few "outlier features", values much bigger than the rest, that matter a lot for performance. If you squash them into the same scale as everything else, you lose them. Their fix kept those rare outliers in higher precision and did more than 99.9% of the multiplications in 8-bit. They reported no performance loss up to 175 billion parameters, while roughly halving inference memory. ### GPTQ, AWQ and GGUF in plain words You'll see these three names constantly on sites like Hugging Face. Two are methods. One is a file format. **GPTQ** (Frantar and colleagues, 2022) quantizes a model one layer at a time. Instead of rounding each weight on its own, it rounds a weight, measures the error that causes on a small sample of real text, and nudges the not-yet-rounded weights to compensate. Think of splitting a restaurant bill: if one person rounds down, someone else rounds up so the total still adds up. The paper quantized a 175B model to 3 or 4 bits in about four GPU hours. At 4 bits, the 175B model lost only 0.03 perplexity (a measure of how surprised the model is by real text; lower is better), while plain rounding lost 2.2, enough to make it worse than a 13B model at full precision. At 3 bits, plain rounding collapsed and GPTQ held up, especially on the larger models. **AWQ** (Activation-aware Weight Quantization, Lin and colleagues at MIT, 2023) starts from the observation that not all weights matter equally. Protecting only about **1%** of the most important weights greatly reduces the error. The clever part is how it finds them: by looking at which input channels carry large values when the model actually runs on sample text, not at the weights themselves. Rather than storing those weights at higher precision (awkward for hardware), it scales them up before rounding, so they effectively get finer resolution. It needs no retraining. It won a best paper award at MLSys 2024. **GGUF** is not a quantization method. It's a **file format** from the GGML project, the library behind **llama.cpp**, a popular program for running models on ordinary computers. A GGUF file packs everything needed to run a model into one file: the (usually quantized) weights, the tokenizer, and settings. llama.cpp supports quantization levels from 1.5-bit to 8-bit. When you see a file called something like `model-Q4_K_M.gguf`, the "Q4" tells you it's roughly 4-bit. One honest caveat: quantization quality is usually reported with averages like perplexity and benchmark scores. Those can hide specific weaknesses. A quantized model might be fine at chat and noticeably worse at maths or code, or in less common languages. If those matter to you, test them directly. --- ## Distillation: a big model as a teacher Quantization keeps the same model and stores it more cheaply. **Distillation** trains a different, smaller model to behave like a big one. The classic paper is "Distilling the Knowledge in a Neural Network" by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, published in 2015. (Hinton credits earlier work by Rich Caruana and colleagues for the basic idea.) Their problem: big models, or groups of models averaged together, worked best, but were too slow to deploy. Could a small model learn what the big one knew? Their key insight was about **what** the small model should learn from. Normally, a model learns from **hard labels**: this picture is a BMW, full stop. But a trained big model knows more than the right answer. It gives a probability for every option. The paper's own example: an image of a BMW might have a tiny chance of being mistaken for a garbage truck, but that's still far more likely than mistaking it for a carrot. Those small probabilities tell you how the big model sees the world: cars look a bit like trucks, and nothing like vegetables. So in distillation, the small "student" learns to match the big "teacher's" full probability spread, which the paper called **soft targets**. To make the small probabilities easier to see, they raised a setting called **temperature** (the same idea as temperature when sampling text), which flattens the spread so the "garbage truck" signal isn't drowned out. The student learns from the teacher's soft targets, usually alongside the true labels.

A hard label only says what's right; a teacher's probabilities also say how wrong each other answer is.

The results were strong. The paper showed it on handwritten digits and on a speech-recognition model used in a heavily used commercial system, where distilling a group of models into one significantly improved the single model. A well-known later example is **DistilBERT** (Hugging Face, 2019): about 40% smaller and 60% faster than the BERT model it learned from, while keeping about 97% of its language-understanding ability on the paper's benchmarks. ### Classic distillation vs training on a big model's outputs Here's a distinction that often gets blurred, including in news coverage. - **Classic distillation** (Hinton-style) needs the teacher's **probabilities** for every possible answer. You usually need full access to the teacher model to get them. - **Training on a big model's outputs** (what Alpaca did) only uses the **text** the teacher wrote. You ask the big model questions, collect its answers, and fine-tune the small model on them as if they were human-written examples. This works through a normal API. People now call both "distillation", and the second is far more common with today's language models. A big example: in January 2025, DeepSeek released six smaller models (1.5B to 70B parameters, built on Qwen and Llama base models) that it described as distilled from its DeepSeek-R1 reasoning model. Its paper says they were made by fine-tuning on **800,000 examples** generated with R1. No probabilities involved, just text. For a 32B model, the paper reported that this beat training the small model directly with reinforcement learning. The analogy: classic distillation is like a student watching the teacher think aloud, including the options they considered and rejected. Training on outputs is like a student copying the teacher's finished homework. Both teach something. The second teaches less. ### What imitation misses That gap was measured. In May 2023, a Berkeley team published "The False Promise of Imitating Proprietary LLMs". They fine-tuned open models of 1.5B to 13B parameters on ChatGPT outputs. Crowd raters thought the imitations were competitive with ChatGPT. But targeted tests showed they closed **little to none** of the real capability gap on tasks the imitation data didn't cover well. Their explanation: **the small models copied ChatGPT's style, not its knowledge**. Confident tone, nice formatting, the right structure. Underneath, the facts and reasoning of a much smaller model. And because the style was convincing, casual human raters missed the difference. This is the most useful thing to remember about distilled or imitation models: **they can sound like the teacher long before they think like it.** Alpaca's 90-to-89 result came from a small, friendly evaluation, exactly the kind that rewards style. --- ## Pruning: cutting out weights A third approach, briefly. **Pruning** removes weights that contribute little, setting them to zero so they can be skipped. SparseGPT (Frantar and Alistarh, 2023) showed that the largest open models of the time, around 175 billion parameters, could have **at least 50%** of their weights removed in one pass, with no retraining and minimal accuracy loss, in under 4.5 hours. It also works together with quantization. The catch: zeros scattered randomly through a model don't automatically make it faster, because hardware likes regular patterns. Pruning pays off most with structured patterns that chips are built to skip (SparseGPT supports some of these), or by removing whole parts, like entire layers or attention heads, which usually needs some retraining to recover quality. In practice, today, quantization and distillation are the workhorses. Pruning is more often an ingredient than the main course. --- ## Running models on laptops and phones Put these together and you get the AI that runs on your own device. A typical recipe for a local model: start from a capable mid-sized model (often itself trained partly on a bigger model's outputs), quantize it to around 4 bits, package it as a GGUF file, and run it with llama.cpp or an app built on it. A 7 to 8 billion-parameter model at 4 bits fits comfortably on a recent laptop. Phones are tighter still. When Apple described its on-device model in June 2024, it said the model was about **3 billion parameters**, compressed with a mix of 2-bit and 4-bit weights averaging **3.7 bits per weight**. Apple said it used small add-on adapters to recover accuracy, and that the result matched the uncompressed model's accuracy on its own measurements. On an iPhone 15 Pro, it reported about 30 tokens per second for generation. Those are Apple's own figures, not independently tested. Why bother running locally? - **Privacy**: your data doesn't leave the device. - **Cost**: no per-token bill once you have the hardware. - **Speed and offline use**: no network round trip. And what you give up: - **Capability**: a 3B or 8B model, however well distilled and quantized, is not a frontier model. It will be weaker at hard reasoning, rare facts, and long tasks. - **Battery and heat** on phones and laptops. - **Your own maintenance**: updates, versions, and testing are now your job. --- ## The licence question: can you train on another model's outputs? This is the part to read carefully, and I'm not a lawyer, so treat it as a map of the questions, not legal advice. **Terms of service.** Many AI providers restrict using their outputs to build competing models. OpenAI's terms of use, at the time of writing (October 2026), list using output "to develop models that compete with OpenAI" among prohibited uses. That's exactly why Stanford flagged Alpaca as research-only. Other providers have their own versions. These are contract terms between you and the provider, and what they mean in practice depends on the exact wording and how they're enforced. **Open model licences.** Open-weight models come with their own licences, and some include rules about using the model or its outputs to train other models. LLaMA's original licence, for example, was non-commercial. Read the licence for the specific model and version you're using, because they change between releases. **Real disputes.** In late January 2025, Bloomberg reported that OpenAI and Microsoft were looking into whether DeepSeek had used distillation from OpenAI's models. OpenAI declined to give details of its evidence, and it was an allegation, not a ruling. It shows the issue is live, and that providers watch for it. **Practical rule:** before you train a model on another model's outputs, check (1) the terms of the model generating the data, (2) the licence of the model you're fine-tuning, and (3) whether your use is commercial. If it matters to your business, ask a lawyer. --- ## What it means in practice If you're choosing or building with smaller models: 1. **Ask "which precision?" when comparing models.** A 4-bit and a 16-bit version of the same model are not identical. Benchmarks should say which one they used. 2. **Consider a bigger model at lower precision before a smaller model at full precision.** Within the same memory budget it's often the better trade, but test both on your own task. 3. **Test the things you care about, not just chat.** Maths, code, your language, your document formats. Quantization and imitation hide their weaknesses in exactly these corners. 4. **Be sceptical of "as good as GPT-whatever" claims for small models.** Check how it was evaluated. Small, friendly evaluations reward style. 5. **Check the licences** before training on another model's outputs or shipping a fine-tuned model commercially. Alpaca's real lesson isn't that $600 buys you a frontier model. It doesn't. It's that the cost of teaching a small model to *behave* well dropped dramatically, and that behaving well and knowing a lot are two different things. --- ## Key ideas - **Model size is mostly memory**: parameters times bytes per parameter. A 7B model is about 14 GB in 16-bit and about 3.5 GB in 4-bit. - **Quantization stores each weight with fewer bits.** 8-bit is near-lossless, 4-bit is usually a good trade, and smart methods like GPTQ and AWQ lose far less than naive rounding. GGUF is a file format, not a method. - **Distillation trains a small student to copy a big teacher.** Hinton's version uses the teacher's full probabilities; most modern "distillation" just fine-tunes on the teacher's text outputs. - **Imitation copies style faster than knowledge.** Small models trained on big-model outputs can sound right while being weaker underneath. - **Pruning removes weights** and can cut half of a huge model with little loss, but it speeds things up only with hardware-friendly patterns. - **On-device AI combines all of these**, trading some capability for privacy, cost and speed. - **Training on another model's outputs raises licence questions.** Check the provider's terms and the model licence first. *Sources: Taori et al., "Alpaca: A Strong, Replicable Instruction-Following Model", Stanford CRFM, 13 Mar 2023 ([crfm.stanford.edu](https://crfm.stanford.edu/2023/03/13/alpaca.html)); Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network", 2015 ([arXiv:1503.02531](https://arxiv.org/abs/1503.02531)); Dettmers et al., "LLM.int8()", NeurIPS 2022 ([arXiv:2208.07339](https://arxiv.org/abs/2208.07339)); Frantar et al., "GPTQ", ICLR 2023 ([arXiv:2210.17323](https://arxiv.org/abs/2210.17323)); Lin et al., "AWQ", MLSys 2024 ([arXiv:2306.00978](https://arxiv.org/abs/2306.00978)); GGML project, GGUF format docs ([github.com/ggml-org/ggml](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)); llama.cpp README ([github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)); Sanh et al., "DistilBERT", 2019 ([arXiv:1910.01108](https://arxiv.org/abs/1910.01108)); DeepSeek-AI, "DeepSeek-R1", Jan 2025 ([arXiv:2501.12948](https://arxiv.org/abs/2501.12948)); Gudibande et al., "The False Promise of Imitating Proprietary LLMs", 2023 ([arXiv:2305.15717](https://arxiv.org/abs/2305.15717)); Frantar and Alistarh, "SparseGPT", 2023 ([arXiv:2301.00774](https://arxiv.org/abs/2301.00774)); Apple Machine Learning Research, "Introducing Apple's On-Device and Server Foundation Models", 10 Jun 2024 ([machinelearning.apple.com](https://machinelearning.apple.com/research/introducing-apple-foundation-models)); OpenAI, Terms of Use ([openai.com](https://openai.com/policies/terms-of-use/)); Fortune, "What is distillation?" on the OpenAI and DeepSeek claims (citing Bloomberg), 29 Jan 2025 ([fortune.com](https://fortune.com/2025/01/29/deepseek-openais-what-is-distillation-david-sacks/)).* *Related: [Long Context and the KV Cache: How AI Reads a Whole Book at Once](https://phamphutai.com/2026/10/20/long-context-and-the-kv-cache/), [Why AI Runs on GPUs](https://phamphutai.com/2026/10/20/why-ai-runs-on-gpus/), and [Generative AI Large Language Model – Train From Pre-trained Model](https://phamphutai.com/2025/06/14/generative-ai-large-language-model-i-fine-tunning-gpt-model/).*