In 2025, a federal court handling a copyright suit against OpenAI ordered the company to preserve user conversations, including chats users had deleted, because the logs might be evidence. Whatever the merits of the underlying case, the order made something concrete that privacy people had been saying abstractly for years: a deleted cloud conversation is a row someone else has been told to keep, and "we delete your data after 30 days" is a policy that a single judge can override. The provider cannot promise otherwise, because the provider does not control the courts it answers to.
That is the backdrop against which local AI got interesting. The open-weight model ecosystem (Llama, Mistral, Qwen, Gemma, and a steady stream of others) has reached the point where a model running entirely on a consumer laptop handles a real share of everyday tasks: summarizing documents, drafting text, answering questions against your own files, transcribing audio. Tools like Ollama, LM Studio, and the llama.cpp project underneath them have turned what used to be a research setup into a download and a double-click.
What Local Inference Actually Removes
When the model runs on your hardware, the prompt never crosses the network. There is no server-side conversation log, which means there is nothing for a retention order to attach to, nothing for a provider breach to expose, and nothing for a future policy change to re-license. The categories of exposure that vanish are exactly the ones you cannot mitigate by being careful:
- Provider retention and legal process. No log exists, so no subpoena, preservation order, or civil discovery request against a provider can reach the conversation.
- Training and review. Cloud providers vary on whether conversations feed model training, and consumer tiers typically require an opt-out where one exists at all. A local model has no upstream to send anything to.
- Account linkage. Cloud chatbot history is tied to an account, which ties every question you have ever asked into one profile. Local inference has no account.
- The workspace problem. Employer-managed AI accounts can expose conversation history to administrators. A model on your own machine answers to nobody's admin console.
It also works offline, which is not a privacy feature so much as a demonstration of one: a tool that functions with the network cable unplugged is making a verifiable claim about where your data goes. That is the strongest form of the argument we made about local-first software generally.
What It Does Not Protect
A local model is software on your computer, and it inherits your computer's problems. The conversation that never reached a cloud log still exists in your chat client's history file on disk, unencrypted unless your disk encryption covers it. If your machine is seized or stolen, local AI history is among the things recovered. If your screen is being captured, the tools we covered in the Windows Recall post will happily archive your private model's answers alongside everything else.
Two subtler points. First, the surrounding software still has network access even when the model needs none: check what your chosen client sends home by way of update checks and telemetry, and firewall it if the answer displeases you. The model file itself is inert weights; the application around it is ordinary software with ordinary habits. Second, model quality is a real gap. The largest cloud models remain meaningfully more capable than what fits in consumer memory, and a wrong answer delivered privately is still a wrong answer. For some tasks that gap does not matter; for others it decides the question.
Apple's approach is worth watching as a third model: on-device inference for most requests, with overflow to servers running audited, stateless software under what Apple calls Private Cloud Compute. It is a stronger promise than a conventional API terms-of-service, and a weaker one than hardware you control. Where it lands for your threat model depends on how much you trust attestation over possession.
A Threat-Model Table Instead of a Verdict
| Concern | Cloud chatbot | Local model |
|---|---|---|
| Provider log / retention order | Exposed, policy-dependent | No log exists |
| Data breach at provider | Possible | Not applicable |
| Device seizure or theft | History reachable via account | History on disk; mitigate with disk encryption |
| Malware / screen capture on your device | Exposed | Exposed equally |
| Answer quality on hard tasks | Stronger | Improving, still behind the frontier |
If You Want to Try It
The practical bar is lower than most people expect. A laptop with 16 GB of memory runs quantized 7-to-12-billion-parameter models at usable speed; Apple Silicon machines and PCs with recent GPUs run larger ones. Ollama and LM Studio both bundle model download, hardware detection, and a chat interface; underneath, they run open-weight models whose files sit in a folder you can point at. Nothing about the setup requires trusting this article: install one, turn off Wi-Fi, and ask it something.
A sensible split for most people is mundane-to-cloud, sensitive-to-local. Use whatever cloud tool your work already uses for tasks you would be comfortable seeing in a discovery filing, and route the questions that touch health, money, law, or other people's confidences to a model whose entire universe is your own RAM. The sorting habit matters more than the tooling. The question to ask before typing something sensitive into any AI is the one this blog keeps returning to: where does this data physically go, and who can be compelled to produce it? For a local model, the answer fits in one word, and that is the whole point.
Related reading: what cloud chatbots retain, keeping your public writing out of training sets, and prompt injection, which applies to local models exactly as much as to cloud ones the moment you feed them untrusted documents.