Run AI Models on Your Own Laptop: A Practical Guide

On an M1 MacBook Pro with 16 GB of memory, a 0.6-billion-parameter model (Qwen3 0.6B, running in Ollama) was asked one question while this post was being written: “What is a GGUF file?”

It answered, instantly and confidently, that GGUF is a standard format used in the computer graphics industry for storing 3D models and textures.

That is wrong. GGUF is the file format the model itself was stored in. The model was reading its own label and didn’t know it.

That exchange sums up local AI. It was free, fast and private, and the answer was worthless. Whether a local model is useful depends on picking the right size for your machine and the right job for the model.

This guide is the practical side: the memory arithmetic, model sizes, file formats, tools, Python, and measurement. The theory of how models get small is in Shrinking AI: Quantization and Distillation; I won’t repeat it here.

Why run a model locally (and why not)

There are five good reasons.

• Privacy. The prompt and the answer never leave your machine. If you’re working with customer data, contracts or medical notes, that can be the whole argument.
• Offline. It works on a plane, in a café with bad wifi, or inside a network that blocks outside APIs.
• Cost. After the hardware, each call is free. For a batch job over 50,000 documents, that changes what you’re willing to try.
• Latency. No network round trip and no queue behind other customers. For short answers from a small model, a local call can start responding faster than a hosted one.
• Learning. You see the moving parts that hosted APIs hide: memory, context length, quantization, speed.

And three honest reasons not to.

• The quality gap. The best hosted models are much larger than anything that fits on a laptop. For hard reasoning, long documents or tricky code, a laptop-sized model will usually lose. The GGUF answer above is what a too-small model looks like.
• Hardware. Memory is the limit, and you can’t add more to most laptops. A model that doesn’t fit either won’t load or runs painfully slowly.
• Maintenance. You own the model, runtime and updates. Nobody rolls out a fix for you.

My advice: treat a local model as one more backend, not a replacement for everything. Use it where privacy, offline or volume matter and the task is narrow enough for a small model: classification, extraction, summaries of short text, first drafts, embeddings. Keep a hosted model for the hard cases.

The memory rule of thumb

A model is a big list of numbers. To run it, those numbers have to sit in memory. So the first question is always: will it fit?

The rule of thumb:

memory needed ≈ parameters × bits per weight ÷ 8, plus the KV cache, plus overhead

The first part is the weights. Divide bits by 8 to get bytes.

The KV cache is the model’s working memory for the conversation: it stores intermediate results for every token in the context so they don’t have to be recomputed. It grows with the length of the context. I explained how it works in Long Context and the KV Cache.

Overhead is everything else: the runtime itself, scratch buffers for the maths, and the parts of the model kept at higher precision.

A worked example: an 8B model

Take Llama 3.1 8B (about 8 billion parameters) in the popular Q4_K_M format. llama.cpp’s own quantization table lists it at 4.89 bits per weight and 4.58 GiB.

Check the arithmetic: 8.03 billion × 4.89 ÷ 8 ≈ 4.9 billion bytes ≈ 4.6 GiB. It matches.

Now the KV cache. Meta’s Llama 3 paper gives the 8B model 32 layers, a model width of 4,096 split across 32 attention heads (so 128 numbers per head), and 8 key/value heads. For each token, the cache holds a key and a value for every KV head in every layer:

2 (key and value) × 32 layers × 8 heads × 128 numbers × 2 bytes (16-bit) = 131,072 bytes, or 128 KiB per token.

• At 8,192 tokens of context: 128 KiB × 8,192 = 1 GiB.
• At 32,768 tokens: 4 GiB.
• At 131,072 tokens: 16 GiB, more than the whole laptop.

So the same model that fits easily at a short context won’t fit at all at its maximum one. Context length is a memory setting, not a free feature.

Overhead, measured

Overhead is the fuzzy part, so here is one measurement. The qwen3:0.6b download from Ollama is 522 MB on disk. Once loaded with Ollama’s default 4,096-token context, ollama ps on the M1 reported 953 MB. The difference is KV cache plus the runtime’s buffers. For a tiny model that nearly doubled the footprint; for an 8B model the share is smaller, but not zero. Leave a gigabyte or two of margin.

Apple Silicon vs NVIDIA vs CPU-only

Where the memory lives changes everything.

Apple Silicon (M1 and later) has unified memory. The CPU and GPU share one pool of RAM. A 32 GB Mac can give a model far more memory than a typical gaming graphics card has. The catch: macOS doesn’t let the GPU use all of it. Metal exposes a limit called recommendedMaxWorkingSetSize. On the 16 GB M1 used for this post it reported 12,713,115,648 bytes (about 11.8 GiB), roughly three quarters of the total. Your Mac’s number will differ, and you also need room for the browser and editor you’re already running.

NVIDIA GPUs have their own VRAM. The model is fast while it fits in VRAM. A card with 12 GB of VRAM is a 12 GB budget, no matter how much system RAM the PC has. When a model is too big, llama.cpp (and Ollama, which uses it) can split layers between the GPU and the CPU. That works, but the CPU half becomes the bottleneck. In Ollama, ollama ps shows this in the PROCESSOR column: 100% GPU is what you want, a split like 48%/52% CPU/GPU means part of the model spilled over.

CPU-only works too. llama.cpp has optimised CPU code for x86 and ARM. It’s slower because generating each token means reading every weight from memory, and system RAM is slower than GPU memory. For small models and short answers, it can still be fine. I covered why memory speed matters so much in Why AI Runs on GPUs.

Picking a model size for your machine

Here is a starting point. This is rough guidance, not a measurement. It assumes a 4-bit format like Q4_K_M (about 4.9 bits, so about 0.61 bytes per parameter) or Q8_0 (about 8.5 bits, so about 1.06 bytes per parameter), a modest context of a few thousand tokens, and that you leave several gigabytes for the operating system and your other apps.

Three adjustments to that table:

1. Add the KV cache for the context you actually need. Long documents cost gigabytes, as the 8B example showed.
2. Mixture-of-experts models only use part of their parameters for each token, which makes them faster, but all the parameters still have to be in memory. Size them by the total, not the active count.
3. Go one size down before going one quant down. Below 4 bits, quality drops faster (the quantization post has the details). A smaller model at Q4_K_M is often a better trade than a bigger one at 2 or 3 bits. Test it on your task.

When in doubt, start small, measure, and move up.

What the file names mean

You’ll mostly meet two formats.

GGUF

GGUF is the single-file format used by llama.cpp and the tools built on it (Ollama and LM Studio among them). One file holds the weights, the tokenizer and the settings. On Hugging Face you’ll see the same model offered in a dozen GGUF variants, named like Model-8B-Instruct-Q4_K_M.gguf.

Reading the suffix:

• Q4, Q5, Q6, Q8: roughly how many bits each weight gets.
• _0 (as in Q8_0, Q4_0): the older, simpler scheme: blocks of weights share one scale.
• _K: the “k-quants” scheme added to llama.cpp in 2023. Weights are grouped into super-blocks of 256, split into smaller blocks, and the per-block scales are themselves quantized. Q4_K works out to about 4.5 bits per weight.
• _S, _M, _L: small, medium, large mixes. In the original k-quants pull request, Q4_K_S uses Q4_K for everything, while Q4_K_M gives some of the most sensitive tensors (half of the attention value and feed-forward output weights) 6-bit precision. That’s why Q4_K_M is slightly bigger than Q4_K_S and a common default.
• IQ (as in IQ3_XXS): the newer “i-quants”, aimed mostly at very low bit counts.

For Llama 3.1 8B, llama.cpp’s table lists Q5_K_M at 5.33 GiB, Q8_0 at 7.95 GiB and unquantized F16 at 14.96 GiB.

A sensible default: Q4_K_M if memory is tight, Q5_K_M or Q6_K if you have room, Q8_0 if you want to be close to the original and can afford it.

MLX (Macs only)

MLX is Apple’s machine-learning framework for Apple Silicon. Its companion package mlx-lm runs and quantizes language models, and the mlx-community organisation on Hugging Face hosts many models already converted, with names like Llama-3.2-3B-Instruct-4bit.

pip install mlx-lm
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit --prompt "Explain a mutex in two sentences."

You can also convert and quantize a Hugging Face model yourself with mlx_lm.convert --model <repo> -q. If you’re on a Mac, it’s worth trying the same model in GGUF and in MLX and measuring both on your own machine, since the speed difference varies.

The tools

Three tools cover most people. All three can expose an OpenAI-compatible HTTP API on localhost. That’s the important part: your app talks to the local model with the same client code it would use for a hosted one, so switching is a config change.

Ollama

Ollama is the easiest start. It wraps llama.cpp, manages downloads, and runs a background server. Install it from ollama.com/download, then:

ollama run qwen3:0.6b        # download if needed, then chat; /bye to quit
ollama pull qwen3:8b         # download without chatting
ollama list                  # what's on disk
ollama ps                    # what's loaded, how big, GPU or CPU

The server listens on http://localhost:11434 and, by default, only accepts connections from the same machine. It has a native API:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:0.6b",
  "messages": [{"role": "user", "content": "Summarise: login fails with 403 on Android only."}],
  "stream": false
}'

And an OpenAI-compatible one at http://localhost:11434/v1/, covering chat completions, completions, embeddings and model listing, among others.

Two defaults matter. Loaded models stay in memory for 5 minutes after the last request (keep_alive). And the context length is small by default: Ollama’s FAQ gives 4,096 tokens, and that’s what ollama ps showed on the M1. If you paste a long document, the start of it may silently fall out of the window. Raise it with num_ctx per request, /set parameter num_ctx in a chat, or the OLLAMA_CONTEXT_LENGTH environment variable, and remember the KV cache arithmetic when you do.

A Modelfile packages a model with your settings, like a Dockerfile for a model:

FROM qwen3:0.6b
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM """You summarise support tickets in two sentences. Never invent details."""

ollama create ticket-summary -f Modelfile
ollama run ticket-summary

Now ticket-summary is a model name any client can call, with the prompt and settings baked in.

llama.cpp

llama.cpp is the engine underneath much of this. Use it directly for full control over flags or a server without a desktop app. The README lists an install script, pre-built binaries, Docker images and source builds. At the time of writing (November 2026), its quick start uses a single llama command:

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The server’s own documentation is written for the llama-server binary, with flags like these:

llama-server -m ./models/model-Q4_K_M.gguf -c 8192 --port 8080

• -m is the GGUF file; -hf <user>/<repo>[:quant] downloads one from Hugging Face instead.
• -c is the context size (by default it’s taken from the model).
• -ngl sets how many layers go on the GPU (auto by default).
• --port: set it explicitly. The documented default at the time of writing is 9931, not the 8080 you’ll see in older tutorials.

It serves /v1/chat/completions, /v1/embeddings and more, plus /health, which returns 503 while the model loads and 200 when it’s ready.

LM Studio

LM Studio is a desktop app with a model browser and chat window, good for exploring before you write code. It runs GGUF models through llama.cpp and, since version 0.3.4 (October 2024), MLX models on Apple Silicon Macs. Its system requirements page lists Apple Silicon on macOS 14 or newer (Intel Macs aren’t supported), Windows on x64 and ARM, and Linux, with 16 GB of RAM recommended.

It has a CLI called lms. lms get <model> downloads a model, and lms server start starts the local server; the docs’ examples use port 1234, with the OpenAI-compatible API at http://localhost:1234/v1.

Calling a local model from Python

All three speak the OpenAI protocol, so the official openai package works unchanged. Ollama ignores the API key, but the client requires one:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")

resp = client.chat.completions.create(
    model="qwen3:0.6b",
    messages=[
        {"role": "system", "content": "Answer in one sentence."},
        {"role": "user", "content": "What is a GGUF file?"},
    ],
    temperature=0,
)
print(resp.choices[0].message.content)

That’s the call that produced the wrong answer at the top of this post. The code was fine; the model was too small for a niche factual question.

To point the same code at LM Studio, change base_url to http://localhost:1234/v1. For llama-server, use the port you started it on. Keep the base URL and model name in config, and your app can switch between local and hosted without a code change.

Ollama also has its own ollama Python package (from ollama import chat) if you want Ollama-specific options, such as think=False to turn off Qwen3’s default reasoning mode.

Everything from Treat the LLM Like an Unreliable API still applies: timeouts, validation, logging. Being on localhost makes a model cheaper, not more trustworthy.

Measuring speed honestly

Speed has two numbers, and they mean different things.

• Prompt processing (prefill): how fast the model reads your input. This dominates when you send long documents.
• Generation (decode): how fast it writes new tokens. This is what you feel when an answer streams.

Ollama’s native API returns both, with durations in nanoseconds:

import requests

def tokens_per_second(model: str, prompt: str) -> tuple[float, float]:
    r = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": model, "prompt": prompt, "stream": False,
              "think": False, "options": {"temperature": 0, "num_predict": 256}},
        timeout=300,
    )
    r.raise_for_status()
    d = r.json()
    prefill = d["prompt_eval_count"] / (d["prompt_eval_duration"] / 1e9)
    decode = d["eval_count"] / (d["eval_duration"] / 1e9)
    return prefill, decode

model = "qwen3:0.6b"
prompt = "Explain what a database index is, for a junior developer."
tokens_per_second(model, prompt)  # warm-up: the first call loads the model
runs = [tokens_per_second(model, prompt) for _ in range(5)]
decode = sorted(r[1] for r in runs)
print(f"decode tokens/s: median {decode[2]:.1f}, range {decode[0]:.1f}-{decode[-1]:.1f}")

One real measurement, for scale. Apple M1 (8-core CPU), 16 GB, macOS 26.3.1, Ollama 0.40.8, qwen3:0.6b (Q4_K_M), fully on the GPU, November 4, 2026. Three batches of five runs gave median generation speeds of 39 to 54 tokens per second (individual runs from 25 to 55), and prompt processing mostly between 750 and 1,900 tokens per second on a short prompt. The machine was busy with other work during the test, which likely explains the spread. A 0.6B model is tiny; an 8B model on the same laptop would be much slower. Don’t use these numbers for anything except a sense of how to report your own.

The habits that make speed numbers worth something:

• Warm up first. The first call includes loading the model from disk.
• Run several times and report the median and range, not your best run.
• Use your real prompt lengths. A 50-token prompt and a 6,000-token prompt are different workloads.
• Write down the exact setup: chip, memory, OS, runtime version, model, quantization, context size.

Quality is the number that matters

Fast and wrong is still wrong. Public leaderboard scores won’t tell you how a small quantized model does on your task (I wrote about why in Benchmarks and Goodhart’s Law).

Build a small test set from your real inputs, with answers you know are right, and run every candidate model through it: the local 4B, the local 8B, the hosted model you’d otherwise use. Evaluating AI Features walks through how. Then the decision is a table, not a feeling. Often the local model wins only on the easy cases, which is still useful if you can route those to it.

Test quantizations too. The same model at Q4_K_M and Q8_0 can differ on maths, code or less common languages even when the averages look similar.

Choosing an open model

Model families change every few months, so check the current options yourself. As of November 2026, these are some of the families in Ollama’s library, with licences I checked on official pages:

• Qwen3 (Alibaba’s Qwen team): sizes from 0.6B to 235B in Ollama’s library. The Qwen3-0.6B model card on Hugging Face lists Apache 2.0, and ollama show qwen3:0.6b --license printed the same.
• gpt-oss (OpenAI): 20B and 120B. Ollama’s page describes a permissive Apache 2.0 licence, says the smaller model can run with as little as 16 GB of memory, and lists the downloads at 14 GB and 65 GB.
• Llama 3.2 (Meta): 1B and 3B text models. The licence is the Llama 3.2 Community License, a custom licence with conditions, including a separate licence requirement for companies above 700 million monthly active users and a “Built with Llama” attribution requirement if you distribute it.
• Gemma 4 (Google DeepMind): E2B, E4B, 12B, 26B (mixture-of-experts) and 31B. Google’s documentation says the weights permit responsible commercial use and links to its Gemma terms; read them before you ship.

Open weights don’t mean no rules. “Open” ranges from standard licences like Apache 2.0 to custom licences with usage policies, attribution rules and user caps. Read the licence on the official model card, not a summary on a download page, before you put a model in a product. Fine-tunes inherit conditions from their base model, so check the base too.

Security notes

A local model reduces one risk (data leaving your machine) and adds a few of its own.

• Download from trusted sources. The official library of your tool, or the model creator’s own Hugging Face organisation. A random re-upload of a popular model can be anything.
• Prefer safe formats. Older PyTorch checkpoints (.bin, .pt) use Python’s pickle format, and Hugging Face’s own documentation warns that loading a pickle can execute arbitrary code. GGUF, MLX and safetensors files store data rather than code, which is a better default. The runtime that parses them is still software, so keep it updated.
• Keep the server on localhost. Ollama binds to 127.0.0.1 by default, and llama-server’s default host is 127.0.0.1. If you change that to reach it from another machine, anyone on the network who can reach the port can use your model and your hardware. Put an authenticating proxy in front, or don’t expose it.
• Treat output as untrusted input, exactly as with a hosted model. Prompt injection works just as well on a local model. If it can call tools, the same rules from Security Holes AI Writes Every Day apply.
• Record what you deployed: model, version, quantization and licence.

Checklist

• Do the arithmetic first: parameters × bits ÷ 8, plus KV cache for your context, plus a gigabyte or two of overhead. Compare it with the memory your GPU can actually use.
• Start with a 4-bit Q4_K_M (or MLX 4-bit) model one size below your limit, measure, then move up.
• Set the context length deliberately. Defaults can be small (Ollama’s FAQ says 4,096 tokens), and long contexts cost gigabytes.
• Use the OpenAI-compatible endpoint and keep the base URL and model name in config, so local and hosted are interchangeable.
• Measure speed properly: warm up, several runs, median and range, real prompt lengths, exact hardware and versions written down.
• Measure quality on your own test set, against the hosted model you’d otherwise use.
• Read the licence on the official model card, download from trusted sources, and keep the server on localhost.

Sources: llama.cpp quantization table, k-quants pull request #1684, llama.cpp README and llama-server docs; The Llama 3 Herd of Models (Meta, 2024); Ollama docs: quickstart, generate API, OpenAI compatibility, Modelfile, FAQ, model library and ollama-python; mlx-lm; LM Studio: developer docs, OpenAI compatibility, system requirements, 0.3.4 release notes; model cards: Qwen3-0.6B, Llama 3.2 1B Instruct, Gemma, gpt-oss on Ollama; Hugging Face on pickle security. All checked November 2026.

Related: Shrinking AI: Quantization and Distillation · Long Context and the KV Cache · Evaluating AI Features