TL;DR
- Auth, rate limiting, input filtering, and audit logs all live on the inbound side of the provider boundary. Once the call is forwarded, the proxy is blind to what happens inside.
- A log records what a system claimed it did. A TEE attestation receipt is a cryptographic artifact tied to the hardware state of the enclave at execution time, verifiable against Intel and NVIDIA public PKI without trusting the vendor.
- If the cache key is too broad, a request from one user can return a response shaped by a prior session's sensitive prompt. Regulated workloads need an explicit answer to that design decision before it becomes a data incident.
- Cloud admins, gateway operators, and provider personnel are inside the system by design. TEE hardware isolation removes their access to plaintext inference content; a privacy policy doesn't.
- Routing a request to a "lower sensitivity" model still sends the prompt to a third-party endpoint. Routing to a TEE model changes where inference physically runs. Same API call, different hardware guarantee.
- IDC projects 65% of regulated enterprise AI deployments will require confidential computing by 2027. A proxy that logs everything but can't prove where compute ran will fail that review, no matter how many provider integrations it supports.
LLM Proxies Sit at the Center of a Gap No One Named Until Now
Gartner projected worldwide AI spending to reach $2.022 trillion in 2026, which puts LLM infrastructure decisions at the center of many procurement conversations right now.
An LLM proxy is a middleware layer that intercepts requests between an application and a model provider before any inference runs. The category grew out of API gateway patterns adapted for token-based workloads: a single endpoint replaces direct provider calls, with routing, logging, and policy enforcement attached. The distinction between proxy, gateway, and router has become a naming dispute more than a functional one. Most production tools do all three jobs from one component, and the labels track vendor heritage more than architecture. "LLM proxy" covers anything from a local developer tool with basic logging to an enterprise control plane governing inference across dozens of teams.
The gap shows up at procurement time, not deployment time. A team installs a proxy, routes traffic through it, and logs everything. Then a compliance review asks not what the proxy logged, but what the execution environment guaranteed. Standard proxies record metadata: request IDs, model names, latency, token counts. They don't control where inference physically runs, who can read memory during execution, or whether the model provider retains anything after the session ends. For teams in defense contracting, financial services, or healthcare, that gap between "we have logging" and "we have verifiable execution" is where AI tools fail security reviews.
This article covers what an LLM proxy controls at the transport and policy layers, where those controls end, and what a team running sensitive workloads needs on top of them. The concrete frame: a platform engineering team deploying AI-assisted development on trading infrastructure. What the proxy covers, where it hands off to the execution environment, and how ORGN Gateway's confidential inference architecture addresses the gap with hardware attestation and sensitivity-based routing.
How an LLM Proxy Controls Request Traffic in Practice
A proxy's value is real and well-defined. The failure mode is treating its scope as larger than it is.
Request Interception and the Policy Enforcement Point
The proxy's core job is intercepting the call before the model provider receives it. At that point, it authenticates the requester, validates the format, applies input filters, and decides whether to forward, block, or rewrite. Auth can run against API keys, SSO tokens, or service identities. Rate limits cap how many requests a team or application can make per minute. Input allowlists reject prompt formats or content patterns before any tokens are spent.
The failure mode appears when teams treat the interception point as equivalent to the execution boundary. The proxy stopped the call at the door. It has no visibility into what happens inside the room. Policy enforcement at the proxy layer is real governance for a real threat surface: unauthorized access, runaway spend, malformed inputs. It doesn't extend to where inference runs or who can read memory during execution.
Token Tracking, Caching, and Cost Visibility
Token counting and semantic caching are standard proxy features. A proxy counts tokens consumed per request and can return a cached response for semantically similar inputs, cutting cost and latency. The proxy's cost visibility depends entirely on what the provider exposes in the response metadata, not on independent measurement.
Caching introduces its own risk surface when cached content includes sensitive context from prior sessions. If the cache key is too broad, a later request from a different user or context can surface a response shaped by an earlier session's prompt. For regulated workloads where prompt content carries data classification requirements, that bleed isn't a corner case. It's a design decision that needs an explicit answer.
Routing Logic Across Multiple Providers
Model routing is where LLM proxies earn the most engineering attention. Rules send a request to one provider or model based on task type, cost threshold, or latency. The problem is that routing decisions made at the proxy layer don't constrain where inference runs at the hardware level. Routing a request to a "lower sensitivity" model on a different provider still sends that prompt to a third-party endpoint. The proxy routes; it doesn't isolate.
Here's a basic Python request to an OpenAI-compatible proxy endpoint, illustrating how the proxy layer sits between the application and the provider:
from openai import OpenAI
client = OpenAI(
base_url="https://api.gateway.orgn.com/v1",
api_key="sk-ollm-your-key-here",
)
response = client.chat.completions.create(
model="vercel_claude_sonnet_4_6", # ZDR path
messages=[{"role": "user", "content": "Generate test scaffolding for auth module"}]
)
print(response.choices[0].message.content)Swapping the model string to near_glm_4_7 routes the same call to a TEE-backed execution environment. The proxy syntax is identical; the hardware guarantee underneath is not. That distinction is what the rest of this article is about.
Where Standard LLM Proxy Controls End
Everything the proxy does is on the request side of the provider boundary. Nothing it does governs the other side.
What the Proxy Doesn't See After Forwarding
Everything the proxy controls exists before the request reaches the provider. The difficult question begins after forwarding, because that's where policy controls stop and execution guarantees become relevant.
This is why sensitivity-based routing isn't simply a provider selection feature. It changes the physical environment where inference occurs, not just the endpoint receiving the request.
Once a proxy forwards a request, its visibility ends at the provider's API surface. The proxy records that a request was sent and a response received. It doesn't know whether the provider logged the prompt, retained it for training, or whether a memory issue exposed it to another tenant during inference.
This isn't a criticism of proxy design. It's a scope boundary. The proxy was built to manage traffic, not enforce execution guarantees inside third-party infrastructure. For regulated workloads, a proxy log doesn't answer what happened in that environment. A log says the call was made. It says nothing about what the provider did with the content.
Attestation vs. Audit Logs: Two Different Claims
An audit log records what a system said it did. An attestation receipt proves what actually ran, at the hardware level.
Standard LLM proxies produce audit logs: structured telemetry showing request metadata, timestamps, token counts, and model identifiers. An attestation receipt, produced only by TEE-backed inference paths, is a cryptographic artifact tied to the enclave's hardware state at execution time. Auditors in financial services and defense increasingly ask for the latter. A proxy log is not a substitute.
The difference matters at the procurement stage. A security team can verify an attestation receipt against Intel and NVIDIA public PKI without trusting the vendor's claims. A proxy log is self-reported by the system it describes. ORGN Scanner produces verifiable attestation records per request on TEE paths, independently verifiable against hardware vendor PKI, as documented in the ORGN Gateway security model.
ORGN Scanner's live messages view showing TEE Attestation status per request. Near-routed requests carry a Verified receipt; Vercel-routed requests carry a ZDR policy tag. The proxy log records the call. Scanner records what the hardware guaranteed.
The Insider Risk Gap Standard Proxies Don't Close
Standard proxy deployments route traffic through infrastructure where operators, including provider personnel, gateway operators, and cloud administrators, potentially have access to plaintext inference content. The proxy enforces policy on the request path; it doesn't restrict what parties further down the chain can read.
For defense contractors working on proprietary algorithms, or healthcare platforms where prompts can include PHI, that exposure is a procurement blocker. The insider risk that proxy-layer controls address is unauthorized access from outside the system. The insider risk they don't address is access by parties who are, by architecture, inside it. TEE execution removes the operator visibility problem: per the ORGN Gateway security model docs, prompts processed inside hardware-isolated Trust Domains are inaccessible to Gateway operators, cloud providers, and hypervisors.
Sensitivity-Based Routing: When the Proxy Needs a Hardware Tier Underneath
The routing problem for regulated teams isn't routing itself. It's what the routing decision actually changes.
Why One Execution Path Doesn't Fit All Workloads
Not every request in a regulated environment carries the same risk. A request to scaffold boilerplate documentation doesn't need the same isolation as one touching authentication logic or patient identifiers. When a proxy routes all traffic through the same execution environment, it either over-protects low-risk requests, adding latency and cost, or under-protects high-risk ones, applying the same risk floor to everything.
Sensitivity-based routing makes the execution path a function of request classification rather than a blanket policy. That's an infrastructure-level decision, not a policy promise. ORGN Gateway's routing logic works at the project and session level: engineers and platform leads configure routing defaults once, so teams don't classify every request individually.
ZDR vs. TEE: Choosing the Right Execution Tier
ORGN Gateway exposes two execution environments behind the same OpenAI-compatible API surface. ZDR models (the vercel_* catalog) run under contractual zero data retention: no logging, no training, no retention. No hardware receipt either. TEE models (near_*, phala_*) run inside Intel TDX confidential virtual machines on NEAR and Phala infrastructure, with memory encrypted at the hardware level and a cryptographic attestation receipt generated per request.
The same API call, with a different model identifier, produces a fundamentally different guarantee. Neither tier retains inference content. Only TEE produces independently verifiable proof that the content never left a hardware-isolated boundary. The ORGN Gateway architecture docs cover the execution environment details, including the control plane separation that prevents Gateway from accessing plaintext content on either path.
ORGN Scanner's team usage view showing mixed-tier routing in a single workspace. Vercel-prefixed models route through the ZDR path; Near-prefixed models route through TEE-backed confidential infrastructure. Token and cost are tracked per model, so classification-level routing decisions have direct observability.
Configuring Project-Level Routing Without Per-Request Overhead
The operational problem with sensitivity routing isn't the concept. It's the implementation overhead. Per-request classification is a bottleneck. ORGN Gateway addresses this at the project and session level: a team can set their entire project to route through the confidential path by default, or configure different defaults for different workflows within the same environment. A healthcare platform processing PHI in prompts routes the whole project through TEE. An internal tooling team working on non-sensitive code scaffolding uses the ZDR path for speed.
The tradeoff is cost: project-wide TEE routing is more expensive than mixed routing. That decision belongs at the data classification level, not the infrastructure team's discretion.
What Confidential LLM Proxy Architecture Covers That Standard Proxies Don't
Hardware-enforced isolation and cryptographic proof of execution are different in kind from policy controls, not just degree.
Hardware-Enforced Memory Isolation During Inference
Intel TDX creates isolated memory regions, called Trust Domains, where inference runs. The host OS can't read the enclave's contents. The hypervisor can't inspect it. Cloud administrators have no access to plaintext execution context. This isn't access control; it's physical separation enforced at the processor level.
ORGN Scanner's Sandboxes view showing TDX-tagged node pools backing the confidential compute layer. These are the Intel TDX infrastructure pools where Trust Domains run. Capacity, zones, and scale are visible; plaintext inference content is not accessible outside these boundaries to the host OS or hypervisor.
Standard cloud sandboxes rely on software-level isolation, which remains accessible to the host. TDX isolation is enforced at the processor, which is what makes the attestation records cryptographically verifiable rather than self-asserted. NVIDIA H100 GPU attestation extends this guarantee to accelerated inference workloads, covering the full computation path.
Cryptographic Attestation and What It Proves to an Auditor
Compliance reviews are rarely blocked because an organization lacks logs. They are blocked because logs describe system behaviour, while auditors increasingly ask for evidence that can be verified independently.
The distinction is subtle but important. A policy tells reviewers what should happen, whereas attestation demonstrates what actually happened during execution using cryptographic evidence rather than vendor assertions.
Each TEE request through ORGN Gateway produces an attestation receipt: a cryptographic artifact proving the specified model ran inside a verified Trust Domain, under expected measurements, at a specific point in time. These records are inspectable in ORGN Scanner, exportable for compliance packages, and independently verifiable against Intel and NVIDIA public PKI without trusting ORGN as an intermediary.
A single TEE request detail in ORGN Scanner showing Intel TDX and NVIDIA GPU attestation both verified. The Intel TDX quote and NVIDIA nonce are cryptographically bound to this specific request, independently verifiable against hardware vendor PKI without trusting ORGN as an intermediary.
What a compliance team can do with this record: attach it to audit packages, present it during FedRAMP or HIPAA reviews, and answer the question "can you prove this code never left a controlled execution environment?" with something other than a policy document. A proxy audit log answers "we logged the call." An attestation receipt answers "we can prove where it ran."
# After a TEE request, the attestation receipt is available in ORGN Scanner
# Verify independently against Intel TDX and NVIDIA public PKI:
# https://scanner.orgn.com -- no ORGN trust required for verification
curl https://api.gateway.orgn.com/v1/chat/completions \
-H "Authorization: Bearer sk-ollm-your-key-here" \
-H "Content-Type: application/json" \
-d '{
"model": "near_glm_4_7",
"messages": [{"role": "user", "content": "Review this auth logic for vulnerabilities"}]
}'
# Response includes attestation metadata; verify receipt in ScannerPicking the Right LLM Proxy Tier for Your Threat Model
Below is a threat model decision:
When a Standard Proxy Is the Right Call
A lightweight proxy is the correct tool when the threat model doesn't require hardware-level execution guarantees. Internal tooling, documentation generation, low-sensitivity code scaffolding, and developer productivity workflows don't need TEE isolation. Adding confidential compute to workloads that don't justify it wastes cost and introduces unnecessary complexity.
The right conditions for a standard proxy: workloads without regulated data, teams without procurement-level attestation requirements, and environments where policy-level zero retention satisfies the compliance posture. ORGN's ZDR path via vercel_* models covers this case: policy-enforced zero retention without hardware overhead, giving moderate-sensitivity workloads strong data handling without the TEE cost.
When the Proxy Alone Isn't Enough
FedRAMP, HIPAA technical safeguards, and financial services data residency requirements don't just require logging. They require demonstrable control over where data is processed and who can access it.
The procurement scenario that surfaces this: a security team asks for attestation records during an AI tooling review. The proxy vendor produces audit logs. The reviewer flags the gap and blocks approval. IDC predicts that by 2027, 65% of enterprise AI deployments in regulated industries will incorporate confidential computing, which signals that procurement teams are already building these requirements into their checklists. ORGN Gateway's TEE path closes that gap with cryptographic proof that's independently verifiable, not self-certified, and inspectable at ORGN Scanner.
Choosing an LLM Proxy Tier Is a Threat Model Decision, Not a Feature List
The question isn't which proxy ships more integrations or has better dashboards. It's what the compliance posture actually requires from the execution layer. A proxy with strong logging, smart routing, and a clean UI still can't tell an auditor where inference ran or who could have read the prompt in transit. Teams that need hardware isolation and attestation records will hit that ceiling regardless of how many features the proxy adds.
This article covered what proxy-layer controls actually govern: request interception, auth, rate limiting, routing, and audit logging on the inbound side of the provider boundary. It covered what they don't govern: the execution environment, insider access, and verifiable proof of compute location. And it covered how sensitivity-based routing with hardware-backed execution tiers changes the architecture for regulated workloads, specifically how ORGN Gateway's TEE and ZDR paths sit behind the same API surface and produce fundamentally different guarantees. If the security review rejected prior AI tooling because no vendor could produce attestation records, that's the specific gap. The decision of whether to close it belongs to the security review, not the procurement checklist.
FAQs
1. What does an LLM proxy actually do to protect a request?
A proxy intercepts calls before they reach a model provider and applies auth, rate limiting, input filtering, and routing rules. It also logs metadata for each request. Protection ends at the provider boundary: the proxy doesn't control what happens inside the execution environment once the call is forwarded.
2. Is an LLM proxy the same as an AI gateway?
The terms overlap in practice. A proxy handles the transport layer: forwarding requests and shaping traffic. A gateway adds policy enforcement, team-level access controls, and compliance controls on top. Most production tools do both jobs from one component, and vendors use the labels based on their product heritage more than a technical distinction.
3. Can an LLM proxy meet HIPAA or FedRAMP compliance requirements?
A proxy contributes to compliance by logging access, enforcing auth, and routing to zero-retention providers. Whether it's sufficient depends on the specific requirement. HIPAA technical safeguards and FedRAMP often require demonstrable control over where PHI is processed, which proxy-layer logging alone doesn't satisfy. That's where hardware-backed execution and attestation records come in.
4. What is the difference between zero data retention and confidential inference in an LLM proxy?
Zero data retention is a contractual guarantee that the provider doesn't log, store, or train on your prompts. Confidential inference is a hardware guarantee: prompts are processed inside a memory-isolated enclave, inaccessible to the provider's own infrastructure. ZDR is a policy claim. Confidential inference produces a cryptographic receipt you can verify independently.