Octopi / Guides / Run AI locally on your Mac
Guide
How to run an AI model locally on your Mac
Yes — a Mac with Apple silicon can run a capable AI model entirely on the machine, with no per-message fees. Here is what you need, how much memory each model size takes, and two ways to set it up: by hand with the free Ollama tool, or with an app that does it for you.
Why run an AI model locally?
A local model is a large language model (LLM) that runs on your own computer instead of a company’s servers. You download it once, and every answer is worked out on your Mac.
Your words stay on your Mac
What you ask and paste is processed on your own computer. No AI provider receives it or keeps it under its terms.
No per-message fees
The software and many models are free to download, and using a model on your disk costs nothing extra.
The model you choose
You pick the model, the size and when to update — or keep several and compare them.
Nobody changes it under you
A model on your disk keeps working the same way if a provider changes its prices, terms or line-up.
The honest limits. Local models are smaller than the largest cloud models, so they are weaker at hard, multi-step reasoning and specialist knowledge. Your Mac’s chip and memory limit speed and model size, and a model can’t look anything up unless the app around it does. Compare the two in local AI vs cloud AI, or see what sovereign AI means for organizations.
What do you need to run a local LLM on a Mac?
An M-series Mac
Any Mac with an Apple M1 chip or a later M-series chip. To check, choose Apple menu ▸ About This Mac: it shows your chip and memory.
Why these Macs work well
The processor and graphics chip share one pool of memory, so the graphics chip can use a loaded model directly. The catch: the model must fit alongside everything else you have open.
Not a good fit
Intel Macs lack unified memory, so local AI tools generally run models on the processor alone — much slower. Some apps, Octopi among them, support Apple silicon only.
Room for the download
Each model is one large download — from about 2 GB for a small model to 20 GB or more for a large one — plus some free space to spare.
How much memory do I need for a local LLM?
Memory matters most. A model’s size is counted in parameters (“7B” is about seven billion), and bigger models need more memory. A rough guide for the 4-bit versions most people run:
| Your Mac’s memory | Model size that fits comfortably | Download, roughly (4-bit) | Example models |
|---|---|---|---|
| 8 GB | Around 3B | About 2 GB | Qwen2.5 3B, Llama 3.2 3B |
| 16 GB | 7B–8B | About 4.5–5 GB | Qwen2.5 7B, Llama 3.1 8B, Mistral 7B |
| 32 GB | Around 14B | About 9 GB | Qwen2.5 14B |
| 64 GB or more | Around 32B and up | About 20 GB for 32B; 40 GB or more for 70B | Qwen2.5 32B; 70B-class models such as Llama 3.1 70B on Macs with the most memory |
A rough guide, not a guarantee. Leave headroom: macOS and your other apps need memory too, and a long conversation or document uses more. Between two rows? Use the lower one.
What is quantization?
A model is mostly a huge list of numbers called weights. Quantization stores each weight with fewer bits — commonly 4 — so the file shrinks to a fraction of its published size and fits in a laptop’s memory. The price is a modest drop in quality, usually a good trade on your own machine. Labels such as “4-bit” or “Q4” mark these versions, and Ollama’s standard downloads are usually 4-bit already.
Option 1: do it yourself with Ollama
Ollama is a free, open-source tool that downloads models and runs them on your Mac, from Terminal or for apps that connect to it.
Install Ollama
Download Ollama for macOS from ollama.com, move it to your Applications folder and open it. It runs in the background, with an icon in the menu bar, and adds the ollama command to Terminal.
Run a model
Open Terminal and type ollama run qwen2.5:7b. The first time, this downloads the model — about 4.7 GB — and then opens a chat prompt. Later runs start from the copy on your disk.
Chat, then tidy up
Type a question and press Return; the answer appears as it is written. Type /bye to leave. Use the commands below to list, add and remove models.
| Command | What it does |
|---|---|
ollama run qwen2.5:7b | Downloads the model if needed, then starts a chat in Terminal |
ollama pull llama3.1:8b | Downloads a model without starting a chat |
ollama list | Lists the models on your Mac and their sizes |
ollama ps | Shows which models are loaded in memory right now |
ollama rm qwen2.5:7b | Deletes a model and frees its disk space |
/bye | Leaves the chat |
Model names come from the library on ollama.com. The part after the colon usually picks the size, such as qwen2.5:3b or qwen2.5:14b.
Ollama isn’t the only way. LM Studio is a desktop app with a point-and-click interface for finding, downloading and chatting with models. llama.cpp is an open-source engine for running models from the command line, and the source of the GGUF model format many local tools use. Pick the interface you prefer.
Our app · one example
Option 2: let an app set it up (how Octopi does it)
If you’d rather skip Terminal, an app can do the setup. Octopi, the free Mac app that publishes this guide, is one example: it turns plain-English tasks into finished spreadsheets, and its chat runs an open-weight model on your Mac through Ollama.
Uses Ollama, or installs it
If Ollama already runs on your Mac, Octopi uses it. If not, it can install a pinned, checksum-verified Ollama release, and stops it when you quit.
A model matched to your memory
The Models window offers four Qwen sizes (below), plus Llama 3.1, Mistral, Gemma 2 and any model already in your Ollama. Its prompts are tuned for Qwen, the default.
Documents, web and CRM
Attach a PDF, Word, RTF or text file (read on your Mac, never uploaded), or have it look things up on the web or in Salesforce, HubSpot or Pipedrive. The conversation isn’t saved.
Online for what you ask
Octopi goes online for look-ups you ask for, the one-time model download, update checks, and usage statistics and error reports (on by default; off in Settings ▸ Privacy).
| Octopi size | Model | Download | Suggested for |
|---|---|---|---|
| Light | qwen2.5:3b-instruct | ~1.9 GB | Below 12 GB of memory |
| Balanced | qwen2.5:7b-instruct | ~4.7 GB | Default from 12 GB |
| Newest | qwen3.5:9b | ~6.6 GB | From 16 GB; noticeably slower |
| Advanced | qwen2.5:14b-instruct | ~9.0 GB | Needs about 32 GB |
Switching between Octopi’s own models removes the one it replaces. The chat refuses models that Ollama runs in its own cloud.
Free, no account, macOS 13 or later on Apple silicon. More: chat with open-source AI models on your Mac.
Tips and troubleshooting
The first reply is slower
The model has to load into memory first, so the first reply after starting — or after a break, if it was unloaded — takes longer.
If your Mac slows to a crawl
A model that doesn’t fit makes macOS swap memory to disk. Close memory-heavy apps; if Activity Monitor ▸ Memory still shows yellow or red Memory Pressure, pick a smaller size.
Let Ollama manage the files
Ollama keeps models in its own folder (by default a hidden .ollama folder in your home folder). Use ollama rm rather than deleting files by hand.
Check that a model is local
Ollama’s library also lists some models that run on Ollama’s own servers, not your Mac. If privacy is the point, choose ones that download and run locally.
Which local model should I pick?
Start with a well-known instruct (chat-tuned) model in the size your memory allows, try it on real tasks, and move up or down a size.
| If you mainly want… | Start with | Why |
|---|---|---|
| General chat and drafting | A 7–8B instruct model: Qwen2.5 7B, Llama 3.1 8B, Mistral 7B or Gemma 2 9B | Solid all-round answers; fits a 16 GB Mac |
| Questions about long documents | The largest model your memory allows | Long text needs extra memory, and bigger models tend to track details better |
| Structured data: tables, fields, JSON | An instruct model of 7B or more | Smaller models drift from a format more often; test on your own examples first |
| The most from an 8 GB Mac | A 3B model | Fine for short questions and quick drafts |
Check the license. Licenses differ even within a family: Qwen2.5 7B and 14B and Mistral 7B use Apache 2.0, while Qwen2.5 3B uses the Qwen Research License, which allows research or evaluation use only. Meta’s Llama 3.1 has its own community license with conditions; Google’s Gemma has the Gemma Terms of Use. Read the model card and license of the exact model and size you use. More in open source vs open-weight AI. This is general information, not legal advice.
Questions
Can I run a local LLM on a Mac with 8 GB of memory?
Yes, a small one. A model of around 3B parameters (roughly a 2 GB download) fits and handles short questions and quick drafts. Close memory-heavy apps first, and expect weaker answers on complex tasks.
Is Ollama free?
Yes. Ollama is free, open-source software for running models on your own computer. Many models are free to download too, but each comes with its own license.
Does a local AI model work without the internet?
The model itself, yes: once downloaded, a tool like Ollama runs it with no connection. Downloading needs the internet, and apps built around a model may go online for features such as web search.
Is a local model as good as a cloud AI assistant?
Not for everything. The largest cloud models are stronger at hard reasoning and specialist questions. A good local model handles everyday questions, drafting and summaries, keeps your words on your Mac and has no per-message cost.
How do I delete a model to free disk space?
With Ollama, run ollama list to see your models and their sizes, then ollama rm with the model’s name — for example, ollama rm qwen2.5:7b.
Related
A local model on your Mac, set up for you
Octopi uses or installs Ollama, suggests a model size for your Mac and gives you a chat that runs on it — free, no account, with the cloud strictly optional.
Download Octopi free