- What a private LLM is (and what it is not)
- Do you actually need one?
- The four ways to build a private LLM
- Where it runs: on-premises, private VPC, or hybrid
- What a private LLM actually costs
- Compliance: what an auditor will ask for
- Day 2: the part most guides skip
- A realistic timeline
- Frequently asked questions
- Sources
A private LLM is a large language model deployed inside infrastructure you control, where prompts, outputs, retrieval data, and logs never leave your security boundary. Building one in 2026 rarely means training a model. For most businesses it means choosing an open-weight model, wrapping it in retrieval over your own documents, and deploying it behind your own identity, logging, and retention controls.
That one paragraph is the honest version of a decision that vendors routinely overcomplicate. The rest of this guide walks the full path: whether you need a private LLM at all, the four architecture options and when each one wins, where to run it, what it costs at current prices, what HIPAA-adjacent auditors actually ask for, and what operating one looks like after launch week.
What a private LLM is (and what it is not)
A private LLM keeps the entire inference loop inside an approved boundary: the model weights, the prompts your staff type, the documents retrieval pulls in, the outputs, and the logs of all of it. Nothing transits a third-party API, and no vendor's retention policy applies to your data, because there is no vendor in the request path.
Two things commonly get mislabeled as private LLMs:
- An enterprise plan on a public API is not a private LLM. Enterprise terms improve contractual protections, and for many businesses that is genuinely enough. But your data still leaves your boundary and is processed on someone else's infrastructure under someone else's controls. If your data classification says records cannot leave approved infrastructure, a contract does not change where the bytes go. We cover this trade-off in depth in Private LLM vs ChatGPT Enterprise.
- A self-hosted model is not automatically a private LLM. Downloading an open-weight model and standing it up on a GPU gets you self-hosting. It becomes private AI when identity integration, access control, prompt logging, retention rules, evaluation, and incident procedures exist around it. Auditors do not accept "it runs on our server" as a control.
Do you actually need one?
Be skeptical of anyone whose answer is always yes. The public-API route is cheaper, faster, and operationally simpler, and for non-sensitive workloads it is usually the right call. Our private LLM vs public API comparison walks the decision in detail, but the short version is that a private LLM earns its cost when at least one of these is true:
- A data classification or contract forbids third-party processing. Common in healthcare, legal, financial services, and government work, where the constraint is not preference but obligation.
- Breach economics dominate the math. IBM's 2025 Cost of a Data Breach Report puts the global average breach at $4.44 million, and healthcare breaches at $7.42 million, the costliest industry for the fourteenth consecutive year. Shrinking the surface where sensitive data travels is a direct lever on that exposure.
- Per-seat AI pricing has outgrown usage. Hosted assistant seats priced per user per month scale with headcount, not value. Past a few dozen heavy users, owned inference can cost less than the subscription line item it replaces.
- You need behavior a hosted tool will not give you. Custom retrieval over proprietary data, tool access to internal systems, pinned model versions that do not change under you overnight.
If none of those apply, bookmark this guide and use a public API with good contractual terms. If one does, keep reading.
The four ways to build a private LLM
Every private LLM build is one of four architectures. The right choice is mostly determined by what you need the model to know and how specialized its behavior must be.
| Approach | What it means | When it wins | Relative cost |
|---|---|---|---|
| RAG on an open-weight model | The model stays stock; retrieval feeds it your documents at question time | Your knowledge changes often; answers must cite sources; fastest path to production | Lowest |
| Fine-tuning | Adjusting model weights on your examples | You need consistent style, format, or domain behavior that prompting cannot hold | Moderate |
| Distillation | Training a smaller model to imitate a larger one | You need low-latency or on-device inference at scale | Moderate to high |
| Training from scratch | Building a foundation model | Almost never; frontier-scale budgets only | Extreme |
RAG (retrieval-augmented generation) is the right starting point for the large majority of business deployments. Your documents stay in a vector store inside your boundary, the model reads only the passages relevant to each question, and updating the system's knowledge means updating documents, not retraining anything. It is also the architecture auditors find easiest to reason about, because you can show exactly which sources produced an answer.
Fine-tuning earns its place when output behavior, not knowledge, is the problem: a model that must write in your clinical documentation format every time, or classify tickets against your internal taxonomy. It layers on top of RAG rather than replacing it.
Distillation and from-scratch training are listed for completeness. If someone proposes training a foundation model for a mid-market deployment, they are proposing you fund their research.
Where it runs: on-premises, private VPC, or hybrid
The deployment boundary question is separate from the architecture question, and it is where compliance teams spend their attention.
- Private VPC (most common). The model runs on dedicated cloud infrastructure inside your virtual private cloud, under your IAM, your network controls, and your logging. You get cloud GPU economics and elasticity without a third party in the inference path. For most regulated mid-market teams this is the right default.
- On-premises. The model runs on hardware you own. Chosen when policy demands physical control, when air-gapping is required, or when steady high utilization makes owned GPUs cheaper than rented ones over a multi-year horizon. The operational burden (hardware lifecycle, patching, capacity planning) is real and belongs in the cost model.
- Hybrid. Sensitive workloads stay on the private deployment; non-sensitive workloads use a public API. In practice most organizations land here, because paying private-inference prices for marketing copy is waste. The governance work is drawing the routing line clearly and enforcing it.
Whichever boundary you choose, the components that make it private are the same: identity-integrated access, prompt and output logging, retention rules, retrieval permissions that mirror document permissions, and monitoring.
What a private LLM actually costs
Most guides go vague here. We publish our pricing, so here is the honest structure of the spend.
Infrastructure. A small RAG deployment serving a team can run on a single modest GPU instance or even shared capacity; steady multi-team serving of a larger open-weight model means dedicated GPU capacity at four to five figures per month at current cloud list prices. Utilization drives everything: an idle dedicated GPU is the most expensive way to answer zero questions, which is why right-sizing during a scoping phase matters more than any vendor discount.
Build. At CloudNSite, every engagement starts with a $999 Discovery Audit, credited toward the build. It maps the workflow, the data boundary, and the right architecture from the table above before any infrastructure is bought. Implementation starts at $8,000 for a focused deployment, $12,000 for multi-workflow operations, and $20,000 for business-critical systems, with production delivery in 4 to 8 weeks.
Operations. The line most budgets miss. Model updates, evaluation runs, retrieval index maintenance, access reviews, and monitoring are ongoing work whether you staff it or contract it. Our managed service runs from $1,500 per month depending on tier; the equivalent internal staffing is a meaningful fraction of an engineer. A private LLM without an operations plan is a compliance finding with a launch date.
The comparison that matters is not private versus free. It is private-LLM total cost versus the sum of per-seat subscriptions, integration workarounds, and the risk exposure of sensitive data in third-party processing. For teams where clauses 1 or 2 above apply, that comparison usually closes.
Compliance: what an auditor will ask for
A private LLM does not make you compliant by existing. It makes compliance *achievable* by putting every control surface inside your boundary. For HIPAA-adjacent deployments, the architecture should be built assuming you will one day answer these requests:
- Who accessed the system, and when? Identity-integrated access logs, not a shared login.
- What did the model see and produce? Prompt and output logging with defined retention, stored inside the boundary.
- What data can retrieval reach? Document-level permissions that mirror your source systems, so the model cannot answer from records the asking user could not open.
- Where is the data processed, and who are the subprocessors? With a private deployment the answer is your infrastructure and none, which is the shortest version of that conversation you will ever have.
- What happens on incident? A written procedure that names the system, because "the AI" appearing in an incident report without one is how audits go long.
For structuring this work, the NIST AI Risk Management Framework's four functions (govern, map, measure, manage) are the reference most security teams already speak. None of this is legal advice; it is the checklist that makes the legal conversation short. Our HIPAA-ready architecture work covers how these controls map to healthcare deployments specifically.
Day 2: the part most guides skip
Launch week is the easy part. The gap between a demo and a production private LLM is everything that happens after:
- Evaluation. A fixed test set of real questions with reviewed answers, run on every change: model updates, retrieval index changes, prompt adjustments. Without it, "the model got worse" is a feeling instead of a diff.
- Monitoring. Latency, refusal rates, retrieval hit rates, and cost per query, with alerts, so degradation is a page and not a user complaint.
- Model lifecycle. Open-weight models improve fast. A deployment pinned to a 2024-era model because nobody owns upgrades is quietly falling behind the public tools your staff compare it against. Upgrades should be scheduled, evaluated against the test set, and boring.
- Access reviews. Quarterly review of who can query what, the control auditors ask about most and teams automate least.
This is the operational load the managed-service line in the cost section exists to carry. Whoever builds your private LLM, make them show you their answer for day 2 before you sign for day 1.
A realistic timeline
For a RAG deployment on an open-weight model in a private VPC, 4 to 8 weeks from scoping to production is realistic: boundary and architecture decisions in week one, retrieval pipeline and integration in the middle weeks, evaluation, access control, and logging hardening at the end. On-premises adds hardware lead time. Fine-tuning adds data preparation time, which is almost always underestimated, budget for cleaning and labeling before training.
What should make you skeptical is a proposal that skips scoping and quotes a build in days (it will be a demo, not a deployment), or one that quotes six months for a first workflow (you are funding someone's learning curve).
Frequently asked questions
Can a small business run a private LLM?
Yes, and the economics have improved every year. A focused RAG deployment on a single workflow with a modest open-weight model starts in the four-figure range to build, not an enterprise program. The threshold question is not company size; it is whether the data justifies private processing.
Do I need to train a model on my data?
Usually no. Retrieval-augmented generation gives the model your knowledge at question time without touching its weights, updates instantly when documents change, and shows its sources. Fine-tuning is for behavior, not knowledge, and most deployments never need it.
Is a private LLM automatically HIPAA compliant?
No. Compliance comes from the controls around the model: access management, logging, retention, retrieval permissions, and incident procedures. A private deployment makes those controls yours to implement and prove, which is exactly what makes compliance achievable.
Which model should we use?
The honest answer is that it should be chosen during scoping, against your latency, quality, and hardware constraints, from the current open-weight field, which changes quarterly. Any guide that names a specific model as "the" answer is dated the month it publishes. The architecture in this guide survives model swaps by design.
What breaks first in self-managed deployments?
Evaluation and upgrades. The system launches, works, and then drifts: the model falls behind, retrieval indexes go stale, and nobody notices until users quietly go back to pasting into public tools. Day-2 operations, whether staffed internally or run as a managed service, is what prevents that.
Sources
- IBM, Cost of a Data Breach Report 2025: global average breach cost of $4.44 million; healthcare highest at $7.42 million for the fourteenth consecutive year.
- The HIPAA Journal, Average Cost of a Healthcare Data Breach (2025): corroborates the IBM healthcare figure and the year-over-year decline.
- NIST AI Risk Management Framework: the govern, map, measure, manage structure referenced for AI risk controls.