Run Open LLMs Locally on Your Mac: What Works, Which Models, and What For

Running a capable language model on a laptop stopped being a party trick about eighteen months ago.

In 2026 a 32 GB MacBook runs models that would have been state of the art in 2024, offline, at 30 to 50 tokens a second, for the price of the electricity. The question is no longer whether it works. It is which runtime, which model, for which job.

This guide answers those three questions for Apple Silicon, with a short detour for Intel Macs and Windows. Everything in it was checked in August 2026, which matters: Ollama switched engines in March, Google relicensed Gemma in April, and Qwen shipped a new 27B six days before this was written.

Who This Is For

• Developers who want a private coding assistant or a local API to build against
• Founders and ops leads with data that cannot leave the building, or a cloud bill that keeps growing
• Automation engineers who need to classify, extract, or transcribe thousands of items without per-token pricing
• Anyone with a recent Mac who has heard it can run AI and wants to know what that actually means

What You Will Need

• An Apple Silicon Mac. 16 GB is the floor for useful work, 32 GB is the sweet spot, 64 GB and up opens the large models.
• 20 to 60 GB of free disk per model family you want to keep around.
• Homebrew, and Python if you go the MLX route directly.
• A realistic job in mind. “Chat with a model” is a demo. “Classify 8,000 support tickets overnight” is a workload, and workloads are where local pays off.

The Pattern

Whatever you install, the shape is the same: a runtime loads quantised weights into unified memory and exposes an OpenAI-compatible HTTP endpoint on localhost. Every tool you already use, from Claude Code to a Python script, points at that endpoint instead of a cloud URL.

flowchart LR
  W[Quantised weights] --> R[Runtime]
  R --> E[localhost API]
  E --> C[Coding agent]
  E --> S[Scripts and pipelines]
  E --> U[Chat UI]

Two numbers govern everything downstream. Unified memory decides which models fit. Memory bandwidth decides how fast they run, almost linearly. An M4 Pro at 273 GB/s generates roughly twice as fast as a base M4 at 120 GB/s on the same model; an M4 Max at 546 GB/s doubles it again. GPU core counts matter far less than the spec sheet suggests.

Pick a Runtime

There are four sensible choices on a Mac in 2026 and they are not really competitors. They sit at different altitudes.

Ollama is where to start. It runs as a service, pulls models by name, and exposes three API dialects: its own, OpenAI’s, and since January an Anthropic-compatible one, which is what lets Claude Code run against a local model with one environment variable. The big 2026 change is under the hood: on Apple Silicon it now runs Apple’s MLX framework rather than llama.cpp on larger-memory Macs, and Ollama’s own measurements showed decode speed roughly doubling on a 35B mixture-of-experts model when it switched.

LM Studio is the GUI answer. It runs MLX and GGUF models side by side, has the best model browser in the category, speaks both OpenAI and Anthropic APIs from localhost:1234, and added parallel requests in its 0.4 release. The app is closed source but free for work use; its MLX engine is open on GitHub.

mlx-lm is Apple’s runtime and the substrate LM Studio and Ollama now build on. mlx_lm.server gives you an OpenAI endpoint in one command, mlx-community on Hugging Face hosts around 4,800 pre-quantised models that load with no conversion, and it is typically 30 to 50 percent faster than llama.cpp on the same hardware. Use it directly when you are scripting or when you want to fine-tune with LoRA.

llama.cpp is the engine everything else wraps for GGUF files. Reach for it directly when a model has no MLX port yet, when you need a long context and want to quantise the KV cache to fit it, or when you are on Windows or Linux, where its CUDA, ROCm, and Vulkan backends are the equivalents of Metal.

A new generation of single-binary servers is worth knowing about. mlx-serve is a 7 MB Zig binary that runs both MLX and GGUF, serves OpenAI, Anthropic, and Ollama APIs on one port, ships a menu-bar app with a built-in agent loop and MCP support, and has one-command launchers for Claude Code, OpenCode, and Aider. oMLX took a different bet, tiering the KV cache between RAM and SSD so that a coding agent’s 15,000-token system prompt is cached across restarts rather than re-processed every turn. Both are six months old, fast-moving, and mostly single-maintainer. Good for a developer’s laptop; give them a year before they hold up a client deliverable.

# Ollama: pull a 27B generalist and talk to it over the OpenAI API
brew install ollama
ollama pull qwen3.6:27b
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6:27b","messages":[{"role":"user","content":"Summarise this in one line: ..."}]}'

# MLX directly: same thing, Apple's runtime, a different port
pip install "mlx-lm[server]"
mlx_lm.server --model mlx-community/Qwen3.6-27B-4bit --port 8080

What Fits in Your Memory

The arithmetic is simple enough to do in your head. At 4-bit quantisation a model needs about 0.55 to 0.6 GB per billion parameters including overhead, at 8-bit about 1.1 GB. Leave a quarter of your RAM for macOS and the context window.

flowchart TD
  M[Unified memory] --> A{How much?}
  A -->|16 GB| B[8-9B dense or gpt-oss-20b]
  A -->|32 GB| C[27-32B dense at 4-bit]
  A -->|64 GB| D[70B dense or 80B MoE at 4-bit]
  A -->|128 GB+| E[120B-class and 70B at 8-bit]
  B --> K[Minus KV cache for context]
  C --> K
  D --> K
  E --> K

Speeds are from independent measurements on M3 Max to M5 Max machines and vary by runtime; take them as bands, not promises. The llama.cpp project keeps a maintained table of every M-series chip if you want your exact machine.

One cell deserves emphasis. Mixture-of-experts models change the economics. Qwen3.6-35B-A3B has 35 billion parameters but activates 3 billion per token, so it needs the memory of a 35B model and runs at the speed of a 3B one. On a 32 GB Mac that is the difference between 20 tokens a second and 100. For batch jobs, MoE is almost always the right shape.

Which Models Are Actually Good

The honest ranking changes monthly. As of 20 August 2026, these are the models worth your disk space, all with permissive licences unless noted.

General work on 32 GB: the 27 to 31B class. Qwen3.6-27B (Apache-2.0, April) reports 77 percent on SWE-bench Verified, which is above many dedicated coding models. Qwen3.8-27B shipped on 14 August with native image and video input and a 262K context; the vendor says it beats its own larger hosted model on coding and office tasks, and independent benchmarks do not exist yet. Gemma 4 31B (April) is the highest-ranked open model on the LMArena text leaderboard that fits a 32 GB laptop, and Google moved the whole Gemma 4 family to Apache-2.0, which removes the licence friction that kept earlier Gemmas out of commercial work.

16 GB: gpt-oss-20b and Qwen3.5-9B. OpenAI’s gpt-oss-20b ships natively in a 4-bit float format that fits in 16 GB, is trained for tool calling, and is the best reasoning model at that size. Qwen3.5-9B is the stronger all-rounder for multilingual text.

Coding. Devstral Small 2 (24B, Apache-2.0) reports 68 percent on SWE-bench Verified and is explicitly built to run on a 32 GB Mac. Qwen3-Coder-30B-A3B is the MoE option at the same memory. On 64 GB, Qwen3-Coder-Next 80B-A3B. In practice the Qwen 27B generalists now match or beat the dedicated coders, so if you only keep one model, keep a generalist.

Vision and documents. Qwen3-VL-8B scores 96 on DocVQA, reads 32 languages, and is the right model for invoices, forms, and screenshots on any Mac with 16 GB. The Qwen3.5 and later generalists and every Gemma 4 size are natively multimodal, so one 27B covers chat and vision.

Speech. NVIDIA’s Parakeet via parakeet-mlx transcribes a 68-minute recording in about a minute on an M3 and beats Whisper on English and 24 other European languages. mlx-whisper or whisper.cpp when you need the other 70 languages or diarisation tooling.

Embeddings. nomic-embed-text runs on CPU and is what most local RAG uses. bge-m3 for multilingual corpora, which for Swiss clients means German, French, Italian, and English in one index. Qwen3-Embedding when quality matters more than speed.

Not laptop models, despite the headlines. DeepSeek V4, GLM-5, Kimi K3, Mistral Small 4, and Llama 4 Scout all need 96 GB to 512 GB. They run on a Mac Studio Ultra. They do not run on yours.

What You Can Actually Do

The workloads where local wins are the ones with volume, privacy, or both.

Private document Q&A. Embeddings with nomic-embed-text, a SQLite or LanceDB vector store, and a 27B model for answers. AnythingLLM, LM Studio, and Jan have this built in for non-developers. One caveat from independent testing: long-document accuracy is the first thing quantisation degrades, so use 5-bit or better for the answering model if the documents are long.

Coding assistant. Point Claude Code at Ollama (ollama launch claude, or ANTHROPIC_BASE_URL=http://localhost:11434 with a 32K context minimum), or use OpenCode, Cline, Aider, or Zed against the OpenAI endpoint. Expect the experience of a capable mid-2025 cloud model: excellent for explaining, refactoring, tests, and small features; not yet for hour-long autonomous runs across a large repo. Context size is the usual failure: below 32K tokens, tool calls stop firing because the schemas get truncated.

Structured extraction. Every runtime now supports JSON-schema-constrained output. The schema guarantees the shape; the model’s size determines whether the content is right. This is the workload that most directly replaces an API bill.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

schema = {
  "type": "object",
  "properties": {
    "vendor": {"type": "string"},
    "invoice_number": {"type": "string"},
    "total": {"type": "number"},
    "currency": {"type": "string", "enum": ["CHF", "EUR", "USD"]},
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {"description": {"type": "string"}, "amount": {"type": "number"}},
        "required": ["description", "amount"],
      },
    },
  },
  "required": ["vendor", "invoice_number", "total", "currency", "line_items"],
}

def extract(invoice_text: str) -> dict:
  response = client.chat.completions.create(
    model="qwen3.6:27b",
    messages=[
      {"role": "system", "content": "Extract invoice fields. Return only the JSON object."},
      {"role": "user", "content": invoice_text},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "invoice", "schema": schema}},
    temperature=0,
  )
  return response.choices[0].message.content

Swap the base_url for a cloud provider and the same code runs there. That portability is the whole argument for the OpenAI-compatible convention.

Transcription to action items. Parakeet or Whisper for the audio, a 27B model for the summary and the task list, bge-m3 if you want to search across meetings later. A one-hour meeting costs about a minute of compute and never leaves the machine.

Batch classification. A 35B-A3B MoE at 100 tokens a second classifies ten thousand rows overnight on a 32 GB Mac. This is the job that makes the hardware pay for itself, and it is the job where LLM-powered data pipelines apply unchanged with a different base URL.

Agents with tools. Qwen3.5 and later, gpt-oss, Gemma 4, and Devstral all do tool calling reliably at 27B and up. Keep to three to five tools with sharp descriptions; small models choose the wrong tool more often than they format the call wrong. Building a Python AI agent covers the loop, and it runs against a local endpoint without modification.

Local, Cloud, or Both

Independent measurement of electricity on a Mac Studio over a month puts the cost of output tokens at about 6 to 11 cents per million for MoE models and around 55 cents for a 27B dense model. Hosted small-model APIs charge $1.25 to $5 for the same million. The hardware is the real cost: a 64 to 128 GB Mac at CHF 4,000 to 6,000 pays back against a CHF 300 to 1,500 monthly API bill in four to fifteen months, but only if the workload tolerates open-model quality.

For Swiss and EU clients the privacy argument is often the deciding one. The revised Federal Act on Data Protection and the GDPR both bite on cross-border transfer and on processor contracts. Running inference locally removes an entire category of questions. It is not, on its own, compliance, and it does not excuse the rest of the data-handling work.

flowchart TD
  Q[Request] --> P{Contains PII or bulk?}
  P -->|Yes| L[Local model]
  P -->|No| H{Hard reasoning?}
  H -->|Yes| C[Cloud frontier model]
  H -->|No| L
  L --> G[One gateway, one API shape]
  C --> G

The practical pattern is a hybrid behind one gateway: local for anything carrying personal data or running in bulk, cloud for the hard reasoning on inputs that have already been redacted. Because both speak the same API, the routing rule lives in one place and the application code never changes. A related trade-off shows up in real-time versus batch pipeline architecture: the batch half of most pipelines is exactly the part that moves local first.

Intel Macs and Windows

If you are still on an Intel Mac with a discrete AMD GPU, almost every tool above has moved on without you. The one active project targeting that hardware is ToshLLM, a native app that bundles llama.cpp with hand-written AMD Metal kernels; its author reports an RX 6700 XT going from unusable to around 60 tokens a second on an 8B model. It is GPL, beta, and small, but it is the only option. On Windows and Linux, Ollama and llama.cpp with CUDA are the mature path; ROCm and Vulkan work with more caveats.

Failure Modes to Avoid

• Forgetting the KV cache. A 70B model’s context at 128K tokens needs as much memory as its weights. Set the context you need, not the maximum, and quantise the cache (--cache-type-k q8_0 in llama.cpp) when you need it long.
• Batch jobs on a MacBook Air. It is fanless. After ten minutes of sustained load it throttles 15 to 25 percent. Fine for chat, wrong for overnight work; use a Pro, mini, or Studio.
• Going below 4-bit to fit a bigger model. Quality falls off a cliff at 3-bit and below, and long-document and tool-calling accuracy go first. A smaller model at 5-bit beats a larger one at 2-bit.
• Downloading weights from whoever uploaded first. In May a repository impersonating an OpenAI release hit number one trending on Hugging Face and delivered an infostealer to 244,000 downloaders in under a day. Take weights from the model author, mlx-community, unsloth, bartowski, lmstudio-community, or ggml-org, and treat “uncensored” uploads from new accounts as what they usually are.
• Exposing the API to the network. Ollama has no authentication by default, tens of thousands of instances are reachable from the internet, and a critical unauthenticated memory-leak vulnerability in it was disclosed in June. Keep it bound to 127.0.0.1, put a reverse proxy with auth or Tailscale in front, and patch.
• Trusting a model recommendation older than a quarter. Half the “best local LLM” posts you will find still recommend Llama 3.1 8B and DeepSeek-R1 distills. Check the model card date before you download.

Before and After

What to Build First

1. Install Ollama and pull one 27B generalist for your memory tier. Talk to it for a day.
2. Point one existing script at localhost:11434/v1 with a JSON schema. Measure accuracy against what you were getting from the cloud.
3. Move the first bulk or private workload over: classification, extraction, or transcription.
4. Add the coding assistant with a 32K context and see where it falls short for your repo.
5. Put a gateway in front with one routing rule: PII or bulk goes local, hard reasoning goes to the cloud.
6. Revisit the model choice every quarter. Pin the weights you ship; do not follow the registry’s latest tag into production.

Final Take

A Mac with 32 GB of memory is a serious inference machine in 2026, and the open models that fit it are good enough for most of the work businesses actually need done.

Start with the workload, not the model. Choose the runtime by the API shape your tools already speak, choose the model by what fits your memory at 4-bit or better, and keep the cloud for the problems that genuinely need it.

The electricity is cheap. The decision about what runs where is the part worth getting right.