Table of Contents
What Self-Hosting an LLM Means
Self-hosting an LLM means running the model itself on hardware you control: your workstation, a server in your rack, or a GPU instance in your cloud account. Your prompts and data never leave your boundary in exchange for you owning the inference stack: the model weights, the serving software, the GPU bill, and the operational work.
Three things made this practical for ordinary teams. Open-weight models became genuinely good: Meta alone publishes families from 1B-parameter models that run on a laptop up to frontier-scale releases, gated only by a license acceptance on Hugging Face. Tooling collapsed the setup cost: Ollama, which positions itself as the easiest way to build with open models, gets a local model serving requests in minutes and offers a path from laptop to datacenter-grade hardware as needs grow. And GPU rental became a commodity: an NVIDIA A10 rents on demand for $1.29 per hour, priced by the minute.
What You Can Actually Run in 2026
The open-weight field changes quarterly, so treat specific models as examples of size classes rather than recommendations. The current Meta lineup on Hugging Face illustrates the range:
- Small (1B to 3B parameters). Llama 3.2 class. Runs on a modern laptop or desktop GPU, with quantization and memory deciding what fits. Good for classification, extraction, routing, and simple drafting. This is where "run an LLM on your own machine" stops being a demo and starts being a utility.
- Mid (7B to 17B dense). The workhorse class: Llama-class 8B and Mistral-class 7B instruct models. A single serious GPU serves these for a team. Handles retrieval-augmented answering, summarization, and structured document work well.
- Large (70B and up). Llama 3.3 70B class and bigger, plus mixture-of-experts releases like Llama 4 Scout, which runs 17B active parameters but 109B total, putting its memory needs in this class despite the headline number. Needs one to several datacenter-class GPUs depending on quantization and context length. This is where output quality becomes competitive with hosted commercial models for many business tasks.
- Frontier-scale open weights (100B+). Historical releases reach 405B. Multi-GPU serving with real engineering. Few businesses need to self-host this class; the ones that do know exactly why.
Alongside Meta's lineup, families from Mistral, Qwen, and others compete in every size class. The right pick depends on your task, latency target, and language mix, which is why we scope model selection during discovery rather than naming a permanent winner.
The Hardware Tiers and What They Cost
Using current on-demand rates from Lambda's GPU cloud as a public benchmark (H100 at $4.29 per hour, A100 at $1.99, A10 at $1.29, billed by the minute, no egress fees):
| Tier | Hardware | What it serves | Cost shape |
|---|---|---|---|
| Workstation | Consumer GPU or Apple Silicon you already own | Small models for one user or light internal tools | Electricity |
| Single rented GPU | One A10 or A100 | Mid-size models for a team; small-model production workloads | ~$940/month for a 24/7 A10, ~$1,450/month for a 24/7 A100 at listed rates |
| Serious inference | One or more H100-class GPUs | 70B-class serving, high concurrency, long context | ~$3,100/month per 24/7 H100 at listed rates, scaling with count |
| Owned hardware | GPUs in your rack | Steady high-utilization workloads, air-gap requirements | Capital cost plus power, cooling, and lifecycle |
Two honest notes on that table. First, 24/7 rental is the worst case: workloads that can batch or scale to zero pay for hours used, and by-the-minute billing makes that real. Second, utilization is everything. An idle dedicated GPU is the most expensive way to answer zero questions. The teams for whom self-hosting pays are the ones with steady workload, which is exactly what makes the math predictable.
The line most budgets miss is not hardware: it is operations. Model updates, evaluation runs, monitoring, access reviews, and index maintenance are ongoing work whether you staff it or contract it. That operational reality is why our managed operations lanes exist alongside the builds.
Self-Hosted Is Not Automatically Private
Standing an open-weight model up on a GPU gets you self-hosting. Private AI is what it becomes once identity integration, access control, logging, retention, evaluation, and incident procedures exist around it, and an auditor will not accept "it runs on our server" as a control.
That distinction matters the moment a self-hosted model touches regulated data or production workflows. The gap between the two is architecture and process, not more hardware, and it is the difference we walk through in how to build a private LLM. If your interest in self-hosting started with HIPAA, client confidentiality, or data residency, read that next, because the compliance answer lives in the surrounding system, not the model.
When Self-Hosting Wins
- Data cannot leave your boundary. Contractual, regulatory, or competitive reasons. Self-hosting removes the third party from the inference path entirely instead of managing it with paperwork.
- Steady, high-volume workloads. Document processing, classification, extraction, and internal assistants that run all day. Owned inference costs track workload, not headcount or tokens, and at sustained volume they can undercut both API metering and per-seat subscriptions. We walk the seat-based version of that math in private LLM vs ChatGPT Enterprise.
- Latency and control requirements. No rate limits you did not set, no model deprecations you did not schedule, no prompt or output leaving your logging perimeter.
- Predictable economics. A GPU bill is boring. Token-metered spend on a workload that grows is not.
When the API Wins
Self-hosting is the wrong answer at least as often as it is the right one:
- Low or spiky usage. If the workload cannot keep a GPU meaningfully busy, per-token API pricing is almost always cheaper.
- You need frontier-model quality. The strongest hosted models still lead open weights on hard reasoning. If your workflow lives on that edge, host the workflow, not the model.
- Nobody owns operations. A self-hosted model with no owner degrades silently: stale weights, no evaluation, no monitoring. If you cannot name the owner, buy the API.
- The real problem is workflow, not inference. Most business AI value comes from what surrounds the model: integration, retrieval, review gates, evaluation. Those exist on every path, and they are usually the harder part.
From Weekend Project to Production System
The honest adoption path we see: someone runs a small model with Ollama on a laptop, proves a workflow is possible, and then the business question arrives: what does this cost and look like as a system the company can rely on?
That jump, from a working demo to production serving with access control, evaluation, monitoring, and an owner, is where projects stall. It is also a well-understood build. We deliver it as a defined scope: architecture, deployment in your cloud or on your hardware, integration with the workflow that justified it, and a stabilization window, with published build pricing for defined-scope work from $8,000 (private deployments are scoped by custom proposal) and the operational side covered under managed service.
If you are weighing self-hosting for a real workload, start with the free 30-minute AI Strategy Call. If the workflow holds up, the $999 Current State Assessment hands you the current-state map and a proposed build with pricing, including the GPU and operations math for your actual volume instead of a generic table.
FAQs
What does it mean to self-host an LLM? Running the model on hardware you control instead of calling a vendor's hosted API, which keeps data inside your boundary and puts the serving stack, costs, and operations in your hands.
What hardware do I need to self-host an LLM? It scales with model size: a laptop or desktop GPU for 1B to 3B models, a dedicated A10 or A100 class GPU for dense 7B to 17B models, and one or more H100-class GPUs for 70B-class and mixture-of-experts models.
How much does it cost to self-host an LLM? At Lambda's current listed rates, a 24/7 A10 is roughly $940 per month and an H100 roughly $3,100, with by-the-minute billing making partial-day workloads far cheaper. Budget operations on top; that line is the one most teams miss.
Is a self-hosted LLM the same as a private LLM? No. Self-hosting is where the model runs; private AI adds the access control, logging, retention, and evaluation an auditor actually checks. A model on your server without those controls is just a server.
Is self-hosting an LLM cheaper than using an API? Only with steady utilization: always-on workloads favor owned inference, bursty ones favor the API because an idle GPU still bills. Calculate the crossover for your real volume before committing.
What is the best self-hosted LLM? It changes quarterly, so pick during scoping against your task, latency, and hardware constraints from the current open-weight field. Architecture that survives model swaps matters more than this quarter's leader.
---
Sources
- Meta Llama on Hugging Face. Publishes the current open-weight lineup by size class (Llama 3.2 at 1B and 3B, Llama 3.3 70B, Llama 4 Scout and Maverick MoE models, historical releases to 405B), gated by license acceptance per repository.
- Lambda, "GPU Cloud". Current on-demand GPU pricing used as the public cost benchmark: NVIDIA H100 SXM at $4.29/hour, A100 at $1.99/hour, A10 at $1.29/hour, billed by the minute with no egress fees.
- Ollama. The best-known local serving tool, positioned as the easiest way to build with open models, with a stated path from local execution to datacenter-grade cloud hardware.