Large language models have transformed how businesses handle document processing, customer service, and internal knowledge management. But for organizations in healthcare, financial services, and government, using public AI APIs creates serious compliance challenges.
The Problem with Public LLM APIs
When you send data to commercial AI services, that data leaves your controlled environment. For organizations handling protected health information (PHI), financial records, or classified data, this creates immediate compliance problems.
- HIPAA requires covered entities to maintain control over PHI. Under the HHS guidance on business associates, a covered entity may only disclose protected health information to a third party once it obtains satisfactory assurances, formalized in a Business Associate Agreement, that the data will be safeguarded. Third-party AI processing therefore requires a BAA and may still create audit concerns.
- SOC 2 Trust Service Criteria for confidentiality become harder to demonstrate when data flows to external AI services.
- PCI DSS explicitly restricts where cardholder data can be processed and stored. The PCI Security Standards Council is direct on this point: do not store cardholder data unless it is absolutely necessary, and retention must be strictly limited to documented business, legal, and regulatory needs.
- Government agencies often have data residency requirements that prohibit external processing entirely.
Even with enterprise agreements from AI providers, your data still leaves your environment. Some providers offer data processing agreements and promise not to train on your data, but auditors and compliance officers often prefer seeing data stay internal. Organizations making the move from public APIs to private LLM deployments typically find the transition smoother than expected when a clear architecture is in place from the start.
Architecture Patterns for Compliant AI
VPC Deployment
The most common pattern for cloud-native organizations is deploying open-source LLMs within your own virtual private cloud. Models like Llama 3, Mistral, and Phi run entirely within your AWS, Azure, or GCP environment. Data never crosses network boundaries you do not control.
GPU instances from cloud providers work well here. AWS offers g5 and p4d instances; Azure has NC and ND series; GCP has A2 and A3 instances. For smaller models (7B to 13B parameters), a single GPU instance handles most workloads. Larger models may need multi-GPU deployments.
On-Premises Deployment
Organizations with existing data centers can deploy LLMs on-premises. This requires hardware investment but provides maximum control. NVIDIA's enterprise GPUs (A100, H100) or AMD alternatives can power private AI infrastructure built around your compliance and security requirements.
On-premises deployment makes sense when you already have GPU infrastructure, when cloud egress costs are significant, or when regulatory requirements mandate physical control over computing resources.
Air-Gapped Environments
For the most sensitive applications, defense, intelligence, and certain financial systems, air-gapped deployment isolates AI systems from any external network. Models and data exist in a completely isolated environment with physical access controls.
Key Controls for Compliance
Deploying privately is only part of the equation. Auditors will look for specific controls around your AI systems.
- Audit Logging: Log every interaction with the LLM including prompts, responses, user identity, and timestamps. This creates the audit trail compliance frameworks require.
- Access Controls: Implement role-based access. Not everyone needs access to AI systems that process sensitive data.
- Data Classification: Know what data types can be processed by AI and enforce boundaries. PHI should only flow to systems designed for PHI.
- Model Governance: Document which models you deploy, their versions, and change management processes. Auditors want to see controlled, predictable AI operations. Many regulated deployments anchor governance on the NIST AI Risk Management Framework, which helps organizations manage AI risks across its Govern, Map, Measure, and Manage functions and incorporate trustworthiness into how AI systems are designed and used.
- Encryption: Data at rest and in transit should be encrypted. This applies to model weights, training data, and inference logs.
Getting Started
Start with a clear inventory of use cases and data types. Identify which applications involve sensitive data and prioritize private deployment there. General-purpose tasks with non-sensitive data might use public APIs while regulated workloads run internally.
The infrastructure investment for private LLM deployment has decreased significantly. Cloud GPU instances are available on-demand. Open-source models have closed much of the capability gap with proprietary alternatives. For many organizations, the total cost of private deployment is now comparable to or lower than high-volume API usage.
If you are evaluating AI for regulated workloads, we can help assess your requirements and design a compliant deployment architecture. Contact us for a consultation.
Sources
- National Institute of Standards and Technology, "AI Risk Management Framework," NIST, 2023. Supports the governance approach, including the Govern, Map, Measure, and Manage functions referenced in model governance.
- U.S. Department of Health and Human Services, "Business Associates," HHS.gov. Supports the requirement that covered entities obtain satisfactory assurances via a Business Associate Agreement before disclosing PHI to a third party.
- PCI Security Standards Council, "PCI Data Storage Do's and Don'ts," PCI SSC. Supports the restriction that cardholder data should not be stored unless absolutely necessary, with retention limited to documented business, legal, and regulatory needs.