On-Premise AI Isn't Automatically More Private

TL;DR

• Keeping inference inside your own data center removes the model provider from the request path, but internal administrators, host tooling, GPU access, logs, and backups can still expose plaintext. The privacy question therefore depends on which actor the deployment is meant to exclude.
• A server's physical location and the team operating its inference stack describe different properties. Self-hosted cloud inference gives the customer software control while leaving the cloud provider inside the infrastructure boundary, whereas on-premises deployment adds physical control alongside the responsibility for operating that infrastructure.
• A prompt isn't private just because the model server doesn't retain it. Reverse proxies, tracing systems, crash handlers, memory inspection, GPU diagnostics, snapshots, and centralized logs can capture sensitive data before the model's own retention policy applies.
• Buying GPUs creates only the first line of the on-premises cost model. Power, cooling, networking, serving software, capacity management, security operations, patching, monitoring, and engineering staff determine whether customer-owned inference remains economically sensible at production load.
• Hardware-backed confidential computing addresses a different problem from data-retention policy. TEE execution protects data during computation from specified host-side infrastructure layers, while ZDR controls retention without providing the same hardware isolation or attestation evidence.
• The deployment choice should follow the threat model, not the word "private." If the security requirement includes proving that a particular workload executed inside an isolated environment and that the GPU participated in that execution, the architecture needs an attestation mechanism rather than an administrative statement alone.

Where the Privacy Boundary Really Sits

An inference server inside your own data center can still expose prompts to a root administrator, a monitoring agent, a crash dump, a backup system, or a compromised host. The GPU sitting behind your firewall says where the workload runs. It says nothing about who can read the workload while the model is processing it.

An on-premises LLM is a language model running on infrastructure physically located in facilities controlled by the organization operating the workload. That distinction matters because physical ownership reshapes several trust relationships, but it doesn't eliminate privileged access from the architecture.

The 2025 CNCF Annual Cloud Native Survey found that 25% of organizations hosting generative AI models self-hosted them on-premises, while 27% self-hosted in the cloud and 37% used managed generative AI services through endpoint APIs. On-premises inference is one choice among several, not the default destination for sensitive workloads.

Some practitioners discussing private inference approach the same tradeoff from the cost side. A 2025 LocalLLaMA thread on LLM hosting costs included a B2B team that had deployed a private model for customers and reported a material cost and performance gap compared with commercial APIs.

The productive question isn't simply whether an LLM runs on-premises. It's private from whom, and what technical control keeps the people or infrastructure inside that boundary from reading the data?

This article examines the trust boundary around on-premises inference, separates on-premises from self-hosted deployment, and accounts for the operational cost sitting behind the GPU purchase. It also compares those properties with confidential inference, where hardware-backed isolation changes which infrastructure layers you need to trust.

On-Premise Answers the Wrong Privacy Question

Moving an inference server into a corporate facility removes some external infrastructure dependencies. It doesn't tell you what happens to plaintext during execution. The answer depends on the host, GPU, serving stack, observability layer, backup system, and the people with privileged access to all of them.

Private from the model provider

Suppose an organization downloads an open-weight model, installs a serving runtime, and keeps inference traffic inside its own network. The external model provider no longer receives the prompt because the organization operates the inference endpoint itself.

That property has real value, particularly when the requirement is to keep source code, internal documents, or regulated records outside a third-party API boundary. A provider-side retention policy becomes irrelevant when the provider never receives the request.

Private from infrastructure administrators

A conventional server gives a privileged administrator considerable visibility into the software running on it. Root access, kernel-level controls, debugging interfaces, process inspection, memory acquisition, and host-level monitoring all sit outside the application-level access controls protecting the inference API. An LLM serving process gets no special exemption because its workload contains confidential data.

Prompt content enters application memory. Tokenized input enters the inference process. Runtime state occupies CPU and GPU memory. Depending on the serving stack, caches and request metadata persist for varying periods. If the threat model includes administrators who operate the host, "the server belongs to us" doesn't close the confidentiality question.

Private from the security operations stack

Security tooling introduces another access path. An organization that turns off prompt persistence in its model server can still have an upstream HTTP debugging layer recording request bodies. Endpoint agents capture process information. Crash handlers collect memory after failures. Application tracing systems attach request metadata to transactions.

The model server's retention policy protects only data that reaches its retention boundary. A sensitive prompt can leave that boundary earlier.

ORGN's security model makes a related distinction for confidential inference: TEE execution protects prompts and responses in memory from the host operating system, hypervisor, cloud provider, and Gateway operators, while ZDR relies on policy and contractual controls rather than hardware isolation.

The same question applies to on-premises architecture: which administrators and infrastructure layers sit outside the confidentiality boundary?

The Five Places an On-Premise LLM Can Still Expose Data

Inference confidentiality depends on the full request path, not just where the model files live. A useful review follows the data from the API request through execution and into every system that can observe or retain the workload.

1. Host memory

The host sees more than a running model binary. The serving process receives plaintext input before tokenization. Application logic constructs request objects. Tokenizers transform text into model input. Runtime components maintain state while the request executes. A host administrator with sufficient privilege sits below all of those application controls.

That separation matters most when the organization's threat model includes compromised hosts or privileged insiders. Network encryption doesn't address memory inspection after the request reaches the server. Disk encryption doesn't protect plaintext while the process runs.

2. GPU memory

GPU memory belongs in the threat model. Modern LLM inference moves substantial computation and workload state onto accelerators, and a security design that protects application memory while ignoring accelerator access leaves an important portion of the execution path outside the stated confidentiality boundary.

GPU administration, diagnostics, firmware, drivers, virtualization layers, and host access all become relevant depending on the deployment architecture.

Confidential GPU execution addresses this differently; for example, ORGN's Gateway describes TEE routes that combine CPU isolation with NVIDIA GPU attestation: Intel TDX with NVIDIA H100 confidential compute for some providers, and AMD SEV-SNP with NVIDIA confidential compute for another supported provider.

The relevant property is whether the hardware and execution environment prevent an infrastructure layer outside the trust boundary from reading inference data.

3. Logs and observability

Logging is one of the easiest ways to defeat a carefully constructed retention policy. Consider an application that sends a customer contract to an internal inference endpoint. The model server keeps no prompts, but an upstream reverse proxy records request bodies for debugging. The observability system then ships those records to a centralized logging cluster with a 90-day retention policy. The inference server remains "zero retention," while the organization stores the same data elsewhere.

The right review question is broader than "does the LLM log prompts?" Trace which components receive the request before, during, and after inference, and whether each one has a documented content-handling policy.

ORGN's Confidential Observe interface configures observability with a write-only API key, the monitoring layer ingests events but cannot read them back. That architectural separation is exactly what the logging review question is looking for.

4. Backups and snapshots

Production systems generate sensitive state beyond model weights. Configuration stores hold endpoints and credentials. Databases hold application state. Logging systems hold operational records. VM snapshots can capture filesystem state that administrators later restore or inspect.

Backup operators therefore become part of the security boundary. An architecture that protects inference traffic but copies sensitive application data into a less-restricted backup environment moves the exposure point, not eliminating it.

5. People

Insider risk doesn't require a malicious administrator. A platform engineer investigating a GPU failure needs host access. A security engineer investigating suspicious activity needs forensic access. A system administrator debugging a kernel issue might inspect processes or memory. Those are routine operational activities.

The relevant question is whether those legitimate privileges include access to plaintext inference data. That question separates administrative control from confidentiality guarantees.

"Private From Whom?" Is the Better Threat Model

The phrase "private AI" compresses several independent security properties into one label. A useful architecture review expands that label back into specific actors, access paths, and evidence.

Deployment model and trust boundary

The table shows why "on-premises" and "private" shouldn't be treated as interchangeable.

A company might own the building, servers, network, and model weights while still granting a broad administrator group access to the hosts. Another company might run an LLM on cloud hardware under hardware-backed isolation that prevents the host layer from reading inference memory. The second architecture has less physical ownership and a narrower execution trust boundary.

ORGN's Gateway architecture applies a similar separation between orchestration and execution. The Gateway router authenticates requests and validates model availability, while selected TEE models execute inside hardware-backed Trust Domains and produce attestation artifacts for verification.

On-Premise LLM vs Self-Hosted LLM: They're Not the Same Thing

The two terms describe different deployment dimensions, and collapsing them leads to poor architecture comparisons. Infrastructure location answers one question while operational ownership answers another.

1. On-premise describes location

On-premise describes where the infrastructure runs. A server installed in a company's data center is on-premise. The same inference stack deployed on a public cloud VM isn't on-premise, even if the customer manages every software component. Physical location affects networking, physical access, procurement, power, cooling, and data residency. It doesn't by itself determine who can inspect application memory.

2. Self-hosted describes operation

Self-hosted describes who operates the software and inference stack. A team can self-host vLLM, TensorRT-LLM, or another serving layer on cloud GPUs. The cloud provider owns and operates the physical infrastructure while the customer operates the model-serving software. Self-hosted, but not on-premises. Software ownership and hardware ownership produce different responsibilities.

The four practical combinations

Hybrid deployments add another dimension; a company might keep a smaller model on-premises for workloads requiring physical network isolation while routing other requests to a managed or confidential execution environment. The decision then happens at the workload level rather than through a single organization-wide deployment rule.

The model-selection mechanism in ORGN Gateway illustrates that separation. The client explicitly chooses a model, and the model identifier determines whether the request reaches a TEE or ZDR execution environment.

3. Seeing the distinction in code

A deployment architecture becomes easier to reason about when the execution choice appears explicitly in the request. ORGN's documented OpenAI SDK integration uses the standard Python client with a Gateway-specific base URL and model ID. The following configuration is directly adaptable for testing a TEE model, since near_glm_4_7 is a documented TEE model identifier.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.gateway.orgn.com/v1",
    api_key=os.environ["OLLM_API_KEY"],
)

response = client.chat.completions.create(
    model="near_glm_4_7",
    messages=[
        {"role": "user", "content": "Summarize this internal policy."}
    ],
)
print(response.choices[0].message.content)

The code doesn't make the application "on-premises," but it demonstrates the opposite architectural point: the application integration surface and the execution security boundary are separate decisions.

The Real Cost of an On-Premise LLM Is Not the GPU

GPU pricing produces an easy procurement number, but the GPU doesn't operate an inference service by itself. The useful TCO model follows every resource and person required to keep inference available, secure, patched, monitored, and capacity-appropriate.

1. Hardware

The first layer covers GPU servers, CPUs, RAM, local storage, networking hardware, spare components, and replacement capacity. The right number depends on model size, quantization, context length, concurrency, latency requirements, and workload shape. A single GPU price says very little about those requirements.

2. Data center

Power and cooling become recurring operating costs. Rack capacity and network capacity matter too. High-density GPU systems change a facility's thermal and electrical requirements compared with conventional application servers. A team that already operates GPU infrastructure has an advantage here. A team buying its first inference cluster needs to price the facility work rather than treating the server as a self-contained appliance.

3. Inference software

Model serving introduces another software estate. The organization owns the serving runtime, GPU drivers, deployment configuration, model artifacts, scheduling, health checks, and update process. Changes to one component can affect the others. The serving stack also determines how it batches, queues, routes, and isolates requests.

4. Platform operations

Someone needs to track GPU utilization, request latency, queue depth, errors, memory pressure, and capacity. Those metrics aren't decorative. They determine whether the cluster has enough capacity, whether a model rollout degraded service, and whether the organization is paying for GPUs that spend much of their time idle.

5. Security

Security costs extend beyond network controls. The environment needs identity and privileged-access management, host hardening, vulnerability management, secrets handling, audit logs, backup controls, and incident response. A private network doesn't remove those requirements.

6. People

People become one of the largest recurring inputs once a deployment reaches production. Platform engineers operate the infrastructure. ML infrastructure engineers manage serving and model changes. Security engineers review access and host controls. SREs handle availability. Procurement and facilities teams manage hardware and power.

ORGN's self-hosted gateway TCO analysis draws the same distinction between the visible infrastructure bill and the engineering work required to keep the system running.

ORGN CDE spinning up a confidential sandbox alongside a running agent codebase: the operational reality behind "self-hosted" is that someone owns the environment, the session, and the execution context.

Where Confidential Computing Changes the On-Prem vs Cloud Debate

Confidential computing addresses a different question from physical ownership. The key question is whether sensitive data stays protected during computation from infrastructure layers outside the workload's trust boundary.

From server ownership to execution isolation

A conventional architecture looks roughly like: Application → Host → GPU → Model. The host operating system, hypervisor, administrators, and accelerator stack surround the workload.

A confidential architecture introduces a protected execution boundary: Application → Protected execution environment → CPU/GPU → Model.

That boundary matters because encryption at rest protects stored data and TLS protects traffic in transit. Neither controls plaintext during computation. ORGN's confidential-computing documentation describes the same data-in-use problem and explains how hardware-encrypted memory protects workloads while the CPU processes them.

Extending the boundary to the GPU

CPU-only protection isn't enough for an inference workload that executes on a GPU.

ORGN's attestation documentation describes a TEE architecture that combines Intel TDX with NVIDIA GPU attestation, or AMD SEV-SNP with NVIDIA confidential-compute GPU attestation, for supported infrastructure. The documentation also describes cryptographic binding between CPU and GPU evidence using a shared session nonce.

Attestation changes the evidence model

Administrative policy states how infrastructure operators are supposed to behave. Attestation produces cryptographic evidence about a measured execution environment.

That distinction matters for regulated environments and adversarial threat models, because a reviewer may need more than a written provider commitment. ORGN's documentation states that TEE requests produce attestation artifacts showing that the specified model ran inside a valid Trust Domain and that the execution environment matched expected measurements. You can inspect those artifacts through ORGN Scanner.

The following model-discovery command comes directly from ORGN's integration documentation. Running it before deployment lets the application select a model ID from the live catalog rather than hardcoding one.

curl https://api.gateway.orgn.com/v1/models \
  -H "Authorization: Bearer $OLLM_API_KEY"

Model discovery doesn't establish confidentiality on its own. It gives the application an authoritative catalog from which the team can select a TEE or ZDR model according to the required trust boundary.

Making the Deployment Decision

An on-premises LLM gives an organization physical and operational control over the inference stack, but that control also assigns the organization responsibility for hosts, GPUs, logs, backups, privileged access, patching, monitoring, capacity, and incident response. Self-hosted cloud inference changes the physical ownership model without changing the customer's responsibility for the serving stack, while confidential inference addresses a different problem entirely: protecting data during execution from infrastructure layers outside the trusted workload boundary.

Before choosing a deployment model, define the actor you need to exclude and the evidence you need to prove that exclusion. The hard question is simple: who must be unable to read the prompt while the model is running?

FAQs

Is an on-premises LLM more private than a cloud LLM?

Only when the threat model is limited to external providers or infrastructure outside the organization; if internal administrators, compromised hosts, monitoring systems, or GPU access fall inside the threat model, on-premises placement alone doesn't establish runtime confidentiality.

What is the difference between on-premises and self-hosted LLMs?

On-premises describes where the infrastructure runs. Self-hosted describes who operates the software and inference stack, so a team can self-host an LLM on public cloud GPUs without running it on-premises.

Does self-hosting eliminate the need for confidential computing?

Not when the threat model includes privileged infrastructure access. Self-hosting gives the customer control over the software and deployment, while confidential computing protects data during execution from host-level infrastructure.

Is on-premises AI cheaper than managed inference?

The answer depends on utilization and existing infrastructure. GPU acquisition is only one cost. Power, cooling, networking, serving software, security operations, patching, monitoring, capacity planning, and engineering time all factor in.