Why I Run My Models Locally

Every prompt you send to a hosted LLM leaves your network. That includes the text you type, the files you attach, and whatever context your tools decide to pack around it. For a quick question about a regex that doesn’t matter. For your mailbox, your customers’ tickets or your company wiki, it matters a lot.

That is the main reason I run most of my models on my own hardware with Ollama. This post covers why, and where these models are useful in a self-hosted setup.

The Privacy Argument

With a local model the whole loop happens inside your own infrastructure:

• The data never leaves. Prompts, documents and answers stay on a box you control. No provider logs, no “we may use your inputs to improve our services”, no surprise retention period.
• No third parties in the chain. There is no API provider, no sub-processor list to review, no data center in a jurisdiction you didn’t choose.
• It still works when the internet doesn’t. No outage, rate limit or changed terms of service on the other side can break your workflow.
• No per-token cost. Once the hardware is there, throwing a whole mailbox or a large document archive at a model costs electricity, not an invoice.

What This Means for the GDPR

The hard part of using a cloud LLM under the GDPR is not the model. It is the data flow: personal data is sent to a processor, often outside the EU. That means a data processing agreement, a look at the sub-processors, a transfer impact assessment if the provider is in the US, and the open question of what happens to the data after the request.

A local model removes that whole layer. Nothing is transferred, nobody else processes the data, so there is no processor agreement and no third-country transfer to justify. You can feed the model your mails, contracts, tickets, customer data or HR documents – data you would never paste into a public chatbot – because it is processed exactly where it already lives.

For purely private use that is the end of the story. In a company, the usual GDPR basics still apply – a purpose and a legal basis for processing the data in the first place – but that is the same as for any other internal tool running on your servers. The LLM no longer adds a compliance problem of its own.

Where I Use Local Models

Ollama exposes an OpenAI-compatible API, so almost every tool that can talk to “ChatGPT” can also talk to your own server. In my homelab Ollama runs as a Swarm stack on a GPU node, and several tools use it.

Chat: Open WebUI

Open WebUI is the ChatGPT-style frontend. It runs in the same stack as Ollama, sits behind Traefik and logs users in via Authentik headers, so everyone in the household or team gets their own chat history without a separate password:

frontend:
  image: ghcr.io/open-webui/open-webui:main
  environment:
    OLLAMA_BASE_URL: http://backend:11434
    WEBUI_AUTH_TRUSTED_EMAIL_HEADER: X-authentik-email
    WEBUI_AUTH_TRUSTED_NAME_HEADER: X-authentik-name
    ENABLE_SIGNUP: "false"

Upload a PDF, ask questions about it, and the document never leaves the house.

Coding: opencode

opencode is a coding agent for the terminal that speaks to any OpenAI-compatible endpoint. Point it at Ollama and you have a coding assistant that can read your private repos – including the ones with internal hostnames, network layouts and customer names in them.

"ollama": {
  "npm": "@ai-sdk/openai-compatible",
  "name": "Ollama",
  "options": { "baseURL": "http://<ollama-host>:11435/v1" },
  "models": {
    "qwen3.5:9b-32k": {
      "name": "qwen3.5:9b-32k",
      "limit": { "context": 32768, "output": 4096 }
    }
  }
}

How I use it day to day is the topic of the next post.

Automation: n8n

n8n has Ollama nodes built in. That turns the model into one step of a workflow: summarize incoming mails, extract fields from invoices, tag documents, write a daily digest of monitoring alerts. These are exactly the jobs where the input is full of personal data – and exactly why they should not leave your network.

Your Own Projects: Boxgent

Because the API is just HTTP, adding a model to your own software is a few lines of code. Boxgent, my self-hosted IMAP filter tool, is a good example of why this matters: it works on people’s mailboxes – about as personal as data gets. A local model can work with that data without a single mail leaving the server.

New: System One Decision Models

Not every task needs a chatty model. Often you just want a decision: which folder does this mail belong in, is this ticket billing or technical, is this comment spam?

Since version 0.35 Ollama also runs System One decision models on a separate /v1/systemone endpoint. Instead of generating text, you send a state plus a set of typed questions, and the model returns a probability for each answer:

{
  "model": "nimble",
  "state": {"ticket": "I was charged twice. Please refund the extra payment."},
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this ticket?",
      "criteria": {"billing": "Payments and refunds", "technical": "Bugs and integrations"}
    }
  }
}

Three models are available so far: nimble (9B), tev1 (4B) and tev1:0.8b. They answer in one request, with no reasoning step, and the results come back as structured data instead of text you have to parse. The performance is really good: fast enough to put in front of every incoming mail or ticket, and small enough to run next to the bigger chat models on the same GPU.

If you run Ollama already, give them a try:

ollama pull nimble
ollama pull tev1

Which Model for Which Job?

Local is not the answer to everything. There are five kinds of models, and the deciding factor is almost always the data, not the model:

• Free tiers – the free plans of ChatGPT, Gemini, Claude, Le Chat and friends: no contract, and often your inputs are used for training
• International APIs – Anthropic, OpenAI, Google: the strongest models, hosted mostly in the US
• EU APIs – Mistral, IONOS, Hetzner, and Microsoft Azure or AWS Bedrock in an EU region, if a data processing agreement is in place
• Rented GPU (HaaS) – a GPU server at a European hoster, running Ollama or vLLM yourself
• Local GPU – your own hardware, in your own rack

✅ fine · ⚠️ possible, check your contracts and risk · ❌ don’t · ⭐ best fit

A few notes on the table:

• Free tiers are the product, not the service. Check the privacy settings – some let you opt out of training – but there is no DPA, so anything you paste there should be something you could also post publicly.
• International providers do offer data processing agreements and are certified under the EU-US Data Privacy Framework. Legally you can send personal data there – but it is a transfer to the US, and I keep that for data where I would also be fine reading about it in the news.
• Microsoft and AWS only land in the EU column with an EU region, a signed DPA and the provider’s EU data boundary settings. They are still US companies, so the CLOUD Act question doesn’t go away completely.
• Rented GPUs give you open-weight models you couldn’t run at home, while you still decide what software runs and what gets logged. There is no provider API logging your prompts – but it is still someone else’s hardware.
• Local wins whenever the data is the most sensitive thing in the room.

My rule of thumb: start at the right end of the table and only move left when the model is really not good enough for the task – and the data allows it.

Conclusion

Local models are not as strong as the biggest hosted ones, and I still use hosted models for some tasks. But for anything that touches personal or confidential data, the trade-off is easy: a slightly smaller model that is allowed to see everything is more useful than a brilliant one that is allowed to see nothing.