Can your business run AI privately?
Yes. Open-weight language models can run entirely on hardware you control — your servers, your private cloud, even an air-gapped machine — so no prompt or document ever reaches a third-party API. A capable private deployment starts around $2,000 in hardware for small-team use, and the quality gap against the frontier APIs is far smaller than most people assume for everyday business tasks. The real decision isn’t whether it’s possible; it’s which of three routes fits your data constraints and budget — and whether you need it at all.
Who actually needs this
Three situations genuinely justify private AI. Regulated data: health records, legal matter files, financial documents — anything where sending content to a third party creates a disclosure question you’d rather not have to answer. Proprietary data: pricing models, deal flow, source code, or client lists that are the business — where “the vendor promises not to train on it” isn’t a comfortable place to rest. Contractual restriction: your own client agreements or upstream terms forbid third-party processing, full stop.
Worth saying plainly: the major providers offer enterprise terms — zero data retention, no-training clauses, HIPAA business associate agreements. For many businesses that’s genuinely enough, and it’s cheaper and better than anything you can host. If none of the three situations above is yours, you probably don’t need this page.
The three routes
| Route | Where your data goes | Rough cost | Best for |
|---|---|---|---|
| API with enterprise terms | To the provider — under zero-retention / BAA contract | Per-token; usually the cheapest to start | Best model quality; teams whose constraint is contractual comfort, not hard prohibition |
| Private cloud GPU | Stays inside your cloud tenancy — the model runs on GPUs you rent | Roughly $300–$2,000+/mo depending on GPU class | No hardware to own; scale up and down; data boundary at your VPC |
| On-premise | Never leaves the building; can run fully air-gapped | One-time hardware buy (next section) + electricity | The hardest data constraints; predictable flat cost under heavy use |
What the hardware actually costs
Ballpark figures as of mid-2026 — hardware prices move, but the tiers hold. The honest headline: most businesses overestimate this by an order of magnitude.
| Budget | What it is | What it runs well | Realistic use |
|---|---|---|---|
| ~$1,000–2,500 | A single used 24 GB GPU in a workstation, or a high-memory Mac mini | Small and mid-size open models (roughly 7–14B, quantized larger) | Document Q&A, summarization, and retrieval for a small team |
| ~$4,000–8,000 | A workstation with one current 24–48 GB GPU, or a maxed Mac Studio | Mid-to-large models (roughly 30–70B quantized) | The sweet spot: most SMB workloads, respectable speed, several concurrent users |
| ~$10,000–20,000 | Dual-GPU server or 48 GB workstation cards | Large models at real speed | Heavier concurrency, larger context windows, latency-sensitive workflows |
| $25,000+ | Datacenter-class GPUs | The largest open models, high throughput | Rarely justified at SMB scale — rent this tier in a private cloud first and measure |
This isn’t theory: I run local model inference on a private, firewalled, high-availability cluster of my own. Right-sizing before you buy — matching model size to the actual task instead of buying the biggest thing — is where most of the money is saved.
What you give up — honestly
The frontier API models are better. On the hardest work — long multi-step reasoning, complex code generation, subtle judgment — the gap is real, and pretending otherwise sells private AI dishonestly. But most business AI isn’t that. Answering questions from your own documents, extracting fields from PDFs, classifying and routing intake, drafting from templates, summarizing records: on grounded, well-scoped tasks like these, good open models are close enough that users rarely notice — because the accuracy comes from the retrieval pipeline feeding the model your documents, not from raw model brilliance.
The right way to decide is not a benchmark chart — it’s an evaluation on your actual tasks with your actual documents, run before committing to hardware. That evaluation is part of any deployment I do.
The pattern that usually wins: hybrid
Private AI and API AI aren’t either/or. The deployment that fits most privacy-constrained businesses routes by sensitivity: the private model handles everything touching regulated or proprietary documents, and a frontier API (under enterprise terms) handles general-purpose work where quality matters and the data is benign. You get the strongest model where it’s safe and the hard boundary where it’s not — at a fraction of the cost of forcing everything through either route.
Signs you don’t need self-hosting
- — No regulator, client contract, or trade-secret concern actually forbids third-party processing. Enterprise API terms will be cheaper and better.
- — Usage is light and occasional. The flat-cost advantage of owned hardware only pays under sustained use.
- — Nobody on the team can own an on-prem box’s care and feeding, and there’s no budget for someone who can. (A retainer solves this — but it should be a chosen cost, not a surprise.)
If any of those describe you, the honest advice is: don’t self-host. It’s the same advice you’d get in an audit — where the deliverable includes the “not worth doing” list.
Common questions
Do we have to self-host to be HIPAA compliant?
No. The major AI providers offer business associate agreements and zero-retention terms, and many covered entities use them. Self-hosting removes the third-party disclosure question entirely rather than managing it by contract — a simpler story for auditors and for clients, but not the only compliant route. Compliance always depends on the whole system, not just where the model runs.
Which models can actually run privately?
Open-weight model families — Llama, Mistral, Qwen, Gemma, DeepSeek and their fine-tunes — in sizes from a few billion to hundreds of billions of parameters. The right one depends on the task and the hardware; a well-chosen mid-size model on a single GPU handles most business document work.
Can it run fully offline / air-gapped?
Yes. Once the model weights are downloaded, inference needs no internet connection at all. Fully air-gapped deployments are practical where the data demands it.
What does private AI cost to operate after setup?
Electricity, occasional hardware maintenance, and someone to keep the software current — there are no per-token fees. Cost is essentially flat regardless of usage, which inverts the API model: heavy internal use gets cheaper per query, not more expensive.
How long does a private AI deployment take?
A working stack — model serving, private retrieval over your documents, and integration into one real workflow — is realistically a matter of weeks rather than months, and right-sizing the hardware before buying is a real part of it. Budget for someone to keep it running afterwards; that ongoing ownership is the cost most people underestimate.
Do you build private AI deployments?
Not at the moment. This guide is written as a reference rather than a pitch — if self-hosting genuinely is what you need, you want someone who does this every day, and I'd rather say so than take the engagement. If what you actually need is automation, integration, or a data pipeline, that I do build.