# Run AI Models on Your Own Laptop: A Practical Guide
> How to run open AI models locally: the memory arithmetic, picking a size for your machine, GGUF and MLX, Ollama, llama.cpp, LM Studio, and Python.
- Author: Tai Pham Phu
- Published: 4 Nov 2026
- Tags: AI Engineering, Local AI, Open Models, Quantization, Applied AI
- URL: https://phamphutai.com/2026/11/04/run-ai-models-on-your-laptop/
On an M1 MacBook Pro with 16 GB of memory, a 0.6-billion-parameter model (Qwen3 0.6B, running in Ollama) was asked one question while this post was being written: *"What is a GGUF file?"*
It answered, instantly and confidently, that GGUF is a standard format used in the computer graphics industry for storing 3D models and textures.
That is wrong. GGUF is the file format the model itself was stored in. The model was reading its own label and didn't know it.
That exchange sums up local AI. It was free, fast and private, and the answer was worthless. Whether a local model is useful depends on picking the right size for your machine and the right job for the model.
This guide is the practical side: the memory arithmetic, model sizes, file formats, tools, Python, and measurement. The theory of how models get small is in [Shrinking AI: Quantization and Distillation](https://phamphutai.com/2026/10/20/quantization-and-distillation/); I won't repeat it here.
---
## Why run a model locally (and why not)
There are five good reasons.
- **Privacy.** The prompt and the answer never leave your machine. If you're working with customer data, contracts or medical notes, that can be the whole argument.
- **Offline.** It works on a plane, in a café with bad wifi, or inside a network that blocks outside APIs.
- **Cost.** After the hardware, each call is free. For a batch job over 50,000 documents, that changes what you're willing to try.
- **Latency.** No network round trip and no queue behind other customers. For short answers from a small model, a local call can start responding faster than a hosted one.
- **Learning.** You see the moving parts that hosted APIs hide: memory, context length, quantization, speed.
And three honest reasons not to.
- **The quality gap.** The best hosted models are much larger than anything that fits on a laptop. For hard reasoning, long documents or tricky code, a laptop-sized model will usually lose. The GGUF answer above is what a too-small model looks like.
- **Hardware.** Memory is the limit, and you can't add more to most laptops. A model that doesn't fit either won't load or runs painfully slowly.
- **Maintenance.** You own the model, runtime and updates. Nobody rolls out a fix for you.
My advice: treat a local model as one more backend, not a replacement for everything. Use it where privacy, offline or volume matter and the task is narrow enough for a small model: classification, extraction, summaries of short text, first drafts, embeddings. Keep a hosted model for the hard cases.
---
## The memory rule of thumb
A model is a big list of numbers. To run it, those numbers have to sit in memory. So the first question is always: **will it fit?**
The rule of thumb:
**memory needed ≈ parameters × bits per weight ÷ 8, plus the KV cache, plus overhead**
The first part is the weights. Divide bits by 8 to get bytes.
The **KV cache** is the model's working memory for the conversation: it stores intermediate results for every token in the context so they don't have to be recomputed. It grows with the length of the context. I explained how it works in [Long Context and the KV Cache](https://phamphutai.com/2026/10/20/long-context-and-the-kv-cache/).
**Overhead** is everything else: the runtime itself, scratch buffers for the maths, and the parts of the model kept at higher precision.
### A worked example: an 8B model
Take Llama 3.1 8B (about 8 billion parameters) in the popular `Q4_K_M` format. llama.cpp's own quantization table lists it at **4.89 bits per weight** and **4.58 GiB**.
Check the arithmetic: 8.03 billion × 4.89 ÷ 8 ≈ 4.9 billion bytes ≈ 4.6 GiB. It matches.
Now the KV cache. Meta's Llama 3 paper gives the 8B model 32 layers, a model width of 4,096 split across 32 attention heads (so 128 numbers per head), and 8 key/value heads. For each token, the cache holds a key and a value for every KV head in every layer:
2 (key and value) × 32 layers × 8 heads × 128 numbers × 2 bytes (16-bit) = **131,072 bytes, or 128 KiB per token.**
- At 8,192 tokens of context: 128 KiB × 8,192 = **1 GiB**.
- At 32,768 tokens: **4 GiB**.
- At 131,072 tokens: **16 GiB**, more than the whole laptop.
So the same model that fits easily at a short context won't fit at all at its maximum one. Context length is a memory setting, not a free feature.
The weights are fixed by the model and the quantization; the KV cache is set by how much context you ask for.
### Overhead, measured
Overhead is the fuzzy part, so here is one measurement. The `qwen3:0.6b` download from Ollama is **522 MB** on disk. Once loaded with Ollama's default 4,096-token context, `ollama ps` on the M1 reported **953 MB**. The difference is KV cache plus the runtime's buffers. For a tiny model that nearly doubled the footprint; for an 8B model the share is smaller, but not zero. Leave a gigabyte or two of margin.
---
## Apple Silicon vs NVIDIA vs CPU-only
Where the memory lives changes everything.
**Apple Silicon (M1 and later) has unified memory.** The CPU and GPU share one pool of RAM. A 32 GB Mac can give a model far more memory than a typical gaming graphics card has. The catch: macOS doesn't let the GPU use all of it. Metal exposes a limit called `recommendedMaxWorkingSetSize`. On the 16 GB M1 used for this post it reported **12,713,115,648 bytes (about 11.8 GiB)**, roughly three quarters of the total. Your Mac's number will differ, and you also need room for the browser and editor you're already running.
**NVIDIA GPUs have their own VRAM.** The model is fast while it fits in VRAM. A card with 12 GB of VRAM is a 12 GB budget, no matter how much system RAM the PC has. When a model is too big, llama.cpp (and Ollama, which uses it) can split layers between the GPU and the CPU. That works, but the CPU half becomes the bottleneck. In Ollama, `ollama ps` shows this in the PROCESSOR column: `100% GPU` is what you want, a split like `48%/52% CPU/GPU` means part of the model spilled over.
**CPU-only works too.** llama.cpp has optimised CPU code for x86 and ARM. It's slower because generating each token means reading every weight from memory, and system RAM is slower than GPU memory. For small models and short answers, it can still be fine. I covered why memory speed matters so much in [Why AI Runs on GPUs](https://phamphutai.com/2026/10/20/why-ai-runs-on-gpus/).
---
## Picking a model size for your machine
Here is a starting point. **This is rough guidance, not a measurement.** It assumes a 4-bit format like `Q4_K_M` (about 4.9 bits, so about 0.61 bytes per parameter) or `Q8_0` (about 8.5 bits, so about 1.06 bytes per parameter), a modest context of a few thousand tokens, and that you leave several gigabytes for the operating system and your other apps.
| Memory (RAM on a Mac, VRAM on NVIDIA) | Sensible starting point | The arithmetic (weights only) |
|---|---|---|
| 8 GB | 1B to 4B at 4-bit | 4B × 0.61 ≈ 2.4 GB |
| 16 GB | 7B to 8B at 4-bit; 14B is a squeeze | 8B × 0.61 ≈ 4.9 GB; 14B × 0.61 ≈ 8.6 GB |
| 32 GB | 14B at 4-bit or 8-bit; about 30B at 4-bit | 14B × 1.06 ≈ 14.9 GB; 32B × 0.61 ≈ 19.6 GB |
| 64 GB | about 70B at 4-bit | 70B × 0.61 ≈ 42.9 GB |
| 128 GB | 70B at 8-bit, or larger mixture-of-experts models | 70B × 1.06 ≈ 74 GB |
Three adjustments to that table:
1. **Add the KV cache** for the context you actually need. Long documents cost gigabytes, as the 8B example showed.
2. **Mixture-of-experts models** only use part of their parameters for each token, which makes them faster, but **all** the parameters still have to be in memory. Size them by the total, not the active count.
3. **Go one size down before going one quant down.** Below 4 bits, quality drops faster (the quantization post has the details). A smaller model at `Q4_K_M` is often a better trade than a bigger one at 2 or 3 bits. Test it on your task.
When in doubt, start small, measure, and move up.
---
## What the file names mean
You'll mostly meet two formats.
### GGUF
**GGUF** is the single-file format used by llama.cpp and the tools built on it (Ollama and LM Studio among them). One file holds the weights, the tokenizer and the settings. On Hugging Face you'll see the same model offered in a dozen GGUF variants, named like `Model-8B-Instruct-Q4_K_M.gguf`.
Reading the suffix:
- **Q4, Q5, Q6, Q8**: roughly how many bits each weight gets.
- **_0** (as in `Q8_0`, `Q4_0`): the older, simpler scheme: blocks of weights share one scale.
- **_K**: the "k-quants" scheme added to llama.cpp in 2023. Weights are grouped into super-blocks of 256, split into smaller blocks, and the per-block scales are themselves quantized. `Q4_K` works out to about 4.5 bits per weight.
- **_S, _M, _L**: small, medium, large *mixes*. In the original k-quants pull request, `Q4_K_S` uses `Q4_K` for everything, while `Q4_K_M` gives some of the most sensitive tensors (half of the attention value and feed-forward output weights) 6-bit precision. That's why `Q4_K_M` is slightly bigger than `Q4_K_S` and a common default.
- **IQ** (as in `IQ3_XXS`): the newer "i-quants", aimed mostly at very low bit counts.
For Llama 3.1 8B, llama.cpp's table lists `Q5_K_M` at 5.33 GiB, `Q8_0` at 7.95 GiB and unquantized `F16` at 14.96 GiB.
A sensible default: **`Q4_K_M`** if memory is tight, **`Q5_K_M`** or **`Q6_K`** if you have room, **`Q8_0`** if you want to be close to the original and can afford it.
### MLX (Macs only)
**MLX** is Apple's machine-learning framework for Apple Silicon. Its companion package `mlx-lm` runs and quantizes language models, and the **mlx-community** organisation on Hugging Face hosts many models already converted, with names like `Llama-3.2-3B-Instruct-4bit`.
```bash
pip install mlx-lm
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit --prompt "Explain a mutex in two sentences."
```
You can also convert and quantize a Hugging Face model yourself with `mlx_lm.convert --model -q`. If you're on a Mac, it's worth trying the same model in GGUF and in MLX and measuring both on your own machine, since the speed difference varies.
---
## The tools
Three tools cover most people. All three can expose an **OpenAI-compatible HTTP API** on localhost. That's the important part: your app talks to the local model with the same client code it would use for a hosted one, so switching is a config change.
Swap any box on the left and the app on the right doesn't change, as long as the API stays OpenAI-compatible.
### Ollama
**Ollama** is the easiest start. It wraps llama.cpp, manages downloads, and runs a background server. Install it from [ollama.com/download](https://ollama.com/download), then:
```bash
ollama run qwen3:0.6b # download if needed, then chat; /bye to quit
ollama pull qwen3:8b # download without chatting
ollama list # what's on disk
ollama ps # what's loaded, how big, GPU or CPU
```
The server listens on `http://localhost:11434` and, by default, only accepts connections from the same machine. It has a native API:
```bash
curl http://localhost:11434/api/chat -d '{
"model": "qwen3:0.6b",
"messages": [{"role": "user", "content": "Summarise: login fails with 403 on Android only."}],
"stream": false
}'
```
And an OpenAI-compatible one at `http://localhost:11434/v1/`, covering chat completions, completions, embeddings and model listing, among others.
Two defaults matter. Loaded models stay in memory for **5 minutes** after the last request (`keep_alive`). And the context length is small by default: Ollama's FAQ gives **4,096 tokens**, and that's what `ollama ps` showed on the M1. If you paste a long document, the start of it may silently fall out of the window. Raise it with `num_ctx` per request, `/set parameter num_ctx` in a chat, or the `OLLAMA_CONTEXT_LENGTH` environment variable, and remember the KV cache arithmetic when you do.
A **Modelfile** packages a model with your settings, like a Dockerfile for a model:
```text
FROM qwen3:0.6b
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM """You summarise support tickets in two sentences. Never invent details."""
```
```bash
ollama create ticket-summary -f Modelfile
ollama run ticket-summary
```
Now `ticket-summary` is a model name any client can call, with the prompt and settings baked in.
### llama.cpp
**llama.cpp** is the engine underneath much of this. Use it directly for full control over flags or a server without a desktop app. The README lists an install script, pre-built binaries, Docker images and source builds. At the time of writing (November 2026), its quick start uses a single `llama` command:
```bash
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
```
The server's own documentation is written for the `llama-server` binary, with flags like these:
```bash
llama-server -m ./models/model-Q4_K_M.gguf -c 8192 --port 8080
```
- `-m` is the GGUF file; `-hf /[:quant]` downloads one from Hugging Face instead.
- `-c` is the context size (by default it's taken from the model).
- `-ngl` sets how many layers go on the GPU (`auto` by default).
- `--port`: set it explicitly. The documented default at the time of writing is **9931**, not the 8080 you'll see in older tutorials.
It serves `/v1/chat/completions`, `/v1/embeddings` and more, plus `/health`, which returns 503 while the model loads and 200 when it's ready.
### LM Studio
**LM Studio** is a desktop app with a model browser and chat window, good for exploring before you write code. It runs GGUF models through llama.cpp and, since version 0.3.4 (October 2024), MLX models on Apple Silicon Macs. Its system requirements page lists Apple Silicon on macOS 14 or newer (Intel Macs aren't supported), Windows on x64 and ARM, and Linux, with 16 GB of RAM recommended.
It has a CLI called `lms`. `lms get ` downloads a model, and `lms server start` starts the local server; the docs' examples use port 1234, with the OpenAI-compatible API at `http://localhost:1234/v1`.
---
## Calling a local model from Python
All three speak the OpenAI protocol, so the official `openai` package works unchanged. Ollama ignores the API key, but the client requires one:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
resp = client.chat.completions.create(
model="qwen3:0.6b",
messages=[
{"role": "system", "content": "Answer in one sentence."},
{"role": "user", "content": "What is a GGUF file?"},
],
temperature=0,
)
print(resp.choices[0].message.content)
```
That's the call that produced the wrong answer at the top of this post. The code was fine; the model was too small for a niche factual question.
To point the same code at LM Studio, change `base_url` to `http://localhost:1234/v1`. For llama-server, use the port you started it on. Keep the base URL and model name in config, and your app can switch between local and hosted without a code change.
Ollama also has its own `ollama` Python package (`from ollama import chat`) if you want Ollama-specific options, such as `think=False` to turn off Qwen3's default reasoning mode.
Everything from [Treat the LLM Like an Unreliable API](https://phamphutai.com/2026/10/14/treat-the-llm-like-an-unreliable-api/) still applies: timeouts, validation, logging. Being on localhost makes a model cheaper, not more trustworthy.
---
## Measuring speed honestly
Speed has two numbers, and they mean different things.
- **Prompt processing** (prefill): how fast the model reads your input. This dominates when you send long documents.
- **Generation** (decode): how fast it writes new tokens. This is what you feel when an answer streams.
Ollama's native API returns both, with durations in nanoseconds:
```python
import requests
def tokens_per_second(model: str, prompt: str) -> tuple[float, float]:
r = requests.post(
"http://localhost:11434/api/generate",
json={"model": model, "prompt": prompt, "stream": False,
"think": False, "options": {"temperature": 0, "num_predict": 256}},
timeout=300,
)
r.raise_for_status()
d = r.json()
prefill = d["prompt_eval_count"] / (d["prompt_eval_duration"] / 1e9)
decode = d["eval_count"] / (d["eval_duration"] / 1e9)
return prefill, decode
model = "qwen3:0.6b"
prompt = "Explain what a database index is, for a junior developer."
tokens_per_second(model, prompt) # warm-up: the first call loads the model
runs = [tokens_per_second(model, prompt) for _ in range(5)]
decode = sorted(r[1] for r in runs)
print(f"decode tokens/s: median {decode[2]:.1f}, range {decode[0]:.1f}-{decode[-1]:.1f}")
```
**One real measurement, for scale.** Apple M1 (8-core CPU), 16 GB, macOS 26.3.1, Ollama 0.40.8, `qwen3:0.6b` (Q4_K_M), fully on the GPU, November 4, 2026. Three batches of five runs gave median generation speeds of **39 to 54 tokens per second** (individual runs from 25 to 55), and prompt processing mostly between **750 and 1,900 tokens per second** on a short prompt. The machine was busy with other work during the test, which likely explains the spread. A 0.6B model is tiny; an 8B model on the same laptop would be much slower. Don't use these numbers for anything except a sense of how to report your own.
The habits that make speed numbers worth something:
- **Warm up first.** The first call includes loading the model from disk.
- **Run several times and report the median and range**, not your best run.
- **Use your real prompt lengths.** A 50-token prompt and a 6,000-token prompt are different workloads.
- **Write down the exact setup**: chip, memory, OS, runtime version, model, quantization, context size.
### Quality is the number that matters
Fast and wrong is still wrong. Public leaderboard scores won't tell you how a small quantized model does on **your** task (I wrote about why in [Benchmarks and Goodhart's Law](https://phamphutai.com/2026/10/20/benchmarks-and-goodharts-law/)).
Build a small test set from your real inputs, with answers you know are right, and run every candidate model through it: the local 4B, the local 8B, the hosted model you'd otherwise use. [Evaluating AI Features](https://phamphutai.com/2026/11/04/evaluating-ai-features/) walks through how. Then the decision is a table, not a feeling. Often the local model wins only on the easy cases, which is still useful if you can route those to it.
Test quantizations too. The same model at `Q4_K_M` and `Q8_0` can differ on maths, code or less common languages even when the averages look similar.
---
## Choosing an open model
Model families change every few months, so check the current options yourself. As of November 2026, these are some of the families in Ollama's library, with licences I checked on official pages:
- **Qwen3** (Alibaba's Qwen team): sizes from 0.6B to 235B in Ollama's library. The Qwen3-0.6B model card on Hugging Face lists **Apache 2.0**, and `ollama show qwen3:0.6b --license` printed the same.
- **gpt-oss** (OpenAI): 20B and 120B. Ollama's page describes a **permissive Apache 2.0 licence**, says the smaller model can run with as little as 16 GB of memory, and lists the downloads at 14 GB and 65 GB.
- **Llama 3.2** (Meta): 1B and 3B text models. The licence is the **Llama 3.2 Community License**, a custom licence with conditions, including a separate licence requirement for companies above 700 million monthly active users and a "Built with Llama" attribution requirement if you distribute it.
- **Gemma 4** (Google DeepMind): E2B, E4B, 12B, 26B (mixture-of-experts) and 31B. Google's documentation says the weights permit responsible commercial use and links to its **Gemma terms**; read them before you ship.
**Open weights don't mean no rules.** "Open" ranges from standard licences like Apache 2.0 to custom licences with usage policies, attribution rules and user caps. Read the licence on the official model card, not a summary on a download page, before you put a model in a product. Fine-tunes inherit conditions from their base model, so check the base too.
---
## Security notes
A local model reduces one risk (data leaving your machine) and adds a few of its own.
- **Download from trusted sources.** The official library of your tool, or the model creator's own Hugging Face organisation. A random re-upload of a popular model can be anything.
- **Prefer safe formats.** Older PyTorch checkpoints (`.bin`, `.pt`) use Python's pickle format, and Hugging Face's own documentation warns that loading a pickle can execute arbitrary code. GGUF, MLX and `safetensors` files store data rather than code, which is a better default. The runtime that parses them is still software, so keep it updated.
- **Keep the server on localhost.** Ollama binds to `127.0.0.1` by default, and llama-server's default host is `127.0.0.1`. If you change that to reach it from another machine, anyone on the network who can reach the port can use your model and your hardware. Put an authenticating proxy in front, or don't expose it.
- **Treat output as untrusted input**, exactly as with a hosted model. Prompt injection works just as well on a local model. If it can call tools, the same rules from [Security Holes AI Writes Every Day](https://phamphutai.com/2026/10/22/security-holes-ai-writes-every-day/) apply.
- **Record what you deployed:** model, version, quantization and licence.
---
## Checklist
- **Do the arithmetic first:** parameters × bits ÷ 8, plus KV cache for your context, plus a gigabyte or two of overhead. Compare it with the memory your GPU can actually use.
- **Start with a 4-bit `Q4_K_M` (or MLX 4-bit) model one size below your limit**, measure, then move up.
- **Set the context length deliberately.** Defaults can be small (Ollama's FAQ says 4,096 tokens), and long contexts cost gigabytes.
- **Use the OpenAI-compatible endpoint** and keep the base URL and model name in config, so local and hosted are interchangeable.
- **Measure speed properly:** warm up, several runs, median and range, real prompt lengths, exact hardware and versions written down.
- **Measure quality on your own test set**, against the hosted model you'd otherwise use.
- **Read the licence on the official model card**, download from trusted sources, and keep the server on localhost.
*Sources: [llama.cpp quantization table](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md), [k-quants pull request #1684](https://github.com/ggml-org/llama.cpp/pull/1684), [llama.cpp README](https://github.com/ggml-org/llama.cpp) and [llama-server docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md); [The Llama 3 Herd of Models (Meta, 2024)](https://arxiv.org/abs/2407.21783); Ollama docs: [quickstart](https://docs.ollama.com/quickstart), [generate API](https://docs.ollama.com/api/generate), [OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility), [Modelfile](https://docs.ollama.com/modelfile), [FAQ](https://docs.ollama.com/faq), [model library](https://ollama.com/library) and [ollama-python](https://github.com/ollama/ollama-python); [mlx-lm](https://github.com/ml-explore/mlx-lm); LM Studio: [developer docs](https://lmstudio.ai/docs/developer), [OpenAI compatibility](https://lmstudio.ai/docs/developer/openai-compat), [system requirements](https://lmstudio.ai/docs/app/system-requirements), [0.3.4 release notes](https://lmstudio.ai/blog/lmstudio-v0.3.4); model cards: [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B), [Llama 3.2 1B Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct), [Gemma](https://ai.google.dev/gemma/docs/core), [gpt-oss on Ollama](https://ollama.com/library/gpt-oss); [Hugging Face on pickle security](https://huggingface.co/docs/hub/security-pickle). All checked November 2026.*
*Related: [Shrinking AI: Quantization and Distillation](https://phamphutai.com/2026/10/20/quantization-and-distillation/) · [Long Context and the KV Cache](https://phamphutai.com/2026/10/20/long-context-and-the-kv-cache/) · [Evaluating AI Features](https://phamphutai.com/2026/11/04/evaluating-ai-features/)*