Octopi / Guides / Local AI vs cloud AI
Guide
Local AI vs cloud AI: privacy, cost and quality compared
The short answer: local AI runs the model on your own device, so your data stays with you and each use costs nothing extra — but your hardware limits how capable the model can be. Cloud AI runs on a provider’s servers: you get the strongest models, you pay per use, and your inputs go to that provider under its terms. Many teams combine the two.
What do “local AI” and “cloud AI” mean?
The difference is where the model runs — the program that reads your prompt and writes the answer. Local AI is also called on-device AI, or self-hosted AI when it runs on your own server.
On hardware you control
The model’s weights are downloaded once and run on your laptop, desktop or a server you run yourself. Engines such as Ollama, LM Studio and llama.cpp download and serve open-weight models.
On a provider’s servers
Your prompt goes over the internet to a company that runs the model in its data centers and sends the answer back — through a cloud AI assistant, an API or a feature in another product.
“Local” describes where the model runs, not whether the app goes online: a tool with a local model may still search the web for you. And a model on your own server counts as local too — the data stays on infrastructure you control.
Local AI vs cloud AI at a glance
| Factor | Local AI | Cloud AI |
|---|---|---|
| Privacy | Prompts and documents are processed on your device; no AI provider receives them | Your inputs are handled by the provider under its terms and retention policy |
| Where data goes | Stays on your device or your own server | Over the internet to the provider’s servers, possibly in another country |
| Cost model | No per-use fee; you pay for the hardware and its electricity | Pay per use or by subscription; cost grows with volume |
| Capability on hard tasks | Limited by what fits in memory; smaller models are weaker at hard reasoning | The strongest models available, with no hardware limit on your side |
| Speed | Depends on your chip and the model’s size | Depends on the provider, the model and your connection |
| Setup | Install an engine and download a model of a few gigabytes | Create an account or get an API key; nothing to install |
| Works without internet | The model does, once downloaded; web tasks still need a connection | No — every request needs a connection |
| Compliance paperwork | Lighter: no third-party processor for the model step; your own duties remain | Under GDPR, a provider processing personal data for you needs a processor agreement, and transfers outside the EEA have their own rules |
| Model and version | You choose the model and keep the exact version until you change it | The provider decides which models and versions are offered, and for how long |
| Vendor lock-in | Low: open-weight models and engines can be swapped | Higher when your workflow depends on one provider’s API and features |
Neither column wins everywhere: it depends on the data, the task and how often you run it.
When is local AI the better choice?
Customer, HR or contract data
If the data shouldn’t leave your organization, a local model means no AI provider receives it. For GDPR, that step then involves no third-party processor — though your own duties, such as a lawful basis and data minimization, still apply.
The same task, many times a day
Classifying emails, tagging records or summarizing notes adds up when every call is billed. Locally, the hundredth run costs the same as the first.
Everyday writing
For first drafts, rewrites and summaries of ordinary text, a good small open-weight model is often enough.
Pulling facts out of text
Turning a page into fields — a company’s city, a person’s role, a date — is a narrow job a local model can handle well, especially when each answer is checked against its source.
A budget that doesn’t move
No usage meter, no surprise invoice when a project grows. Your cost is the machine you already own.
Rules about where data is processed
Some organizations must keep processing in a given jurisdiction. Others prefer non-US or self-hosted processing because of laws such as the US CLOUD Act (2018), under which US authorities can require US-based providers to disclose data in their control, including data stored abroad. Running the model yourself settles where that step happens.
More on this: What is sovereign AI? and using AI with customer data under GDPR. This is general information, not legal advice.
When is cloud AI the better choice?
Problems that need the strongest model
Multi-step analysis, tricky code and questions where a small model loses the thread are where the largest cloud models lead.
Breaking a big job into steps
Turning a loose request into a long, reliable plan is demanding; a larger model tends to make fewer mistakes. And a plan can often be made from the task’s description alone, without the data.
A machine without the memory
A model must fit in memory alongside your other apps. Roughly, a 7–8B model at 4-bit needs 4.5–5 GB and a 70B model 40 GB or more. Without that headroom, the cloud is the practical option.
A few times a month
If you rarely use AI, paying per use can cost less than buying a machine with enough memory — and there is nothing to install.
Using the cloud for sensitive work isn’t wrong in itself; it means reading the provider’s terms and, for personal data, signing the right agreements. Just choose it on purpose. For hardware and model sizes, see how to run an AI model locally on your Mac.
The hybrid approach: local by default, cloud on purpose
Many teams don’t pick a side. They run a local model for everyday volume and anything sensitive, and call a cloud model only for specific jobs where it clearly helps — sending as little as possible, such as a description of the task rather than the records.
If you evaluate a tool that mixes the two, look for three things:
Opt-in per feature
Each cloud feature is off until you turn it on, separately. One switch that sends everything is not a real choice.
It says what is sent
Before you agree, the tool tells you exactly which data goes to the provider — and which never does.
It falls back
If the cloud is unavailable or out of budget, work carries on locally and the tool says so, instead of failing or stopping silently.
How to decide: five questions
Ask them per task: the answer often differs from job to job.
| Question | Leans local | Leans cloud |
|---|---|---|
| 1. Would it matter if this data left your device? | Yes — personal, confidential or regulated data | No — public or already-shared information |
| 2. How hard is the task? | Drafting, summarizing, classifying, extracting | Hard multi-step reasoning or long, complex plans |
| 3. How often will you run it? | Daily, or in large batches | Now and then |
| 4. Does your hardware have room? | Memory to spare for the model and your apps (on Apple silicon Macs, CPU and GPU share one pool of unified memory, which suits local models) | Little free memory, or an older machine |
| 5. Do rules or contracts say where data may be processed? | Yes — keep it on your device or your own server | No — but still check the provider’s terms |
Mixed answers are normal, and usually point to a hybrid: local for the sensitive volume, cloud for the hardest steps.
An example
Where Octopi fits
Octopi, the free macOS app this site is about, is one example of the hybrid approach. It turns a plain-English task into a finished spreadsheet, researching companies, people and facts in a real browser and writing each value into your sheet with a clickable source.
An open-weight model on your Mac
The model runs on your Mac through Ollama. The default, Qwen2.5-7B-Instruct, is published under the Apache 2.0 license; Llama 3.1, Mistral and Gemma 2 are offered too, and their licenses vary.
Claude can plan your task
If you turn it on, Claude receives your task sentence and your column names — not your rows.
Claude can read public pages
If you turn it on, Claude reads public web pages for you, loaded in a private, signed-out headless browser.
Your CRM and signed-in sites
Your CRM (Salesforce, HubSpot or Pipedrive) and any site you sign in to are always handled on your Mac.
Each Claude option is off until you turn it on, with its own consent screen. You pay with your own Anthropic key ($0 from Octopi) or with Octopi credits. If Claude is unavailable or your credits run out, Octopi carries on with the local model and says so.
It isn’t an offline app: Octopi goes online to search and read the pages you ask for, use your signed-in CRMs, download the model once and check for updates. Usage statistics and scrubbed error reports are on by default; switch them off in Settings ▸ Privacy. Full breakdown: Sovereign AI on your Mac.
Questions
Is local AI private?
More private for the model step: your prompts and documents are processed on your own device, so no AI provider receives them. But privacy also depends on what else the app sends — usage data, web searches, synced files — so check that too.
Is a local LLM as good as a cloud AI assistant?
For drafts, summaries, classification and extraction, a good small model is often enough. On the hardest reasoning and long, complex planning, the largest cloud models are stronger, and a local model is limited by your machine’s memory.
Is local AI free?
There is no per-use fee, and Ollama and many open-weight models are free to download. You still pay for the hardware, and each model’s license sets what you may do with it — check the model card of the exact model and size.
What are the pros and cons of self-hosted AI?
Pros: your data stays on infrastructure you control, no per-use fees, and you keep the model version you chose. Cons: you provide and maintain the hardware, its memory limits which models you can run, and the most capable models are offered as cloud services.
Does a local model make me GDPR compliant?
Not by itself. Processing on your own device means no third-party processor is involved for that step, but your own duties — a lawful basis, data minimization, purpose limitation — still apply. This is general information, not legal advice.
Does Octopi use local or cloud AI?
Local by default: an open-weight model runs on your Mac through Ollama. Claude is optional, for planning and/or reading public pages, each off until you turn it on with its own consent screen. If Claude is unavailable, Octopi carries on with the local model and says so.
Related
Try local-first AI on your Mac
Download Octopi: an open-weight model on your Mac by default, with Claude strictly optional. Free, no account needed.
Download Octopi free