The Hidden Risk of Shared Infrastructure in AI Development Tools

TL;DR

• GPU hardware costs $25,000-$40,000 per unit, lead times stretch 36-52 weeks, and DDR5 memory prices have surged over 400% since mid-2025, making shared infrastructure the default model for virtually every AI coding assistant and inference API on the market.
• On shared infrastructure, your code and prompts pass through tokenization, inference, and output generation in plaintext memory on hardware the provider controls, accessible to the host OS, hypervisor, and observability tooling at every stage.
• "Do not train" clauses, SOC 2 certifications, and terms of service commitments govern intended behavior but cannot prevent misconfiguration, legal compulsion under the US CLOUD Act, or a breach exposing data that was technically accessible during processing.
• Trusted Execution Environments move the security boundary from software to silicon. Memory inside a TEE is encrypted by the CPU itself, inaccessible to the host even with administrative access, and verifiable through per-request cryptographic attestation.
• ORGN runs every workspace in an Intel TDX-encrypted sandbox by default, backs ORGN model requests with both Intel TDX and NVIDIA GPU Attestation, and gives users full control over data persistence; no prompts, code, or outputs are stored for ORGN confidential model sessions.

AI development tools have made it easy to overlook the infrastructure beneath them. You write a prompt, get code back, and move on. What most developers don't think about, and most vendors don't explain, is where that prompt actually goes, who else is using the same hardware, and what access the provider has to your data while it's being processed.

What Is Shared Infrastructure and Why the Entire AI Industry Runs on It

AI inference runs on GPUs, and right now, GPUs are in a full-blown supply crisis driven almost entirely by AI demand. A single NVIDIA H100, the current standard for AI workloads, costs between $25,000 and $40,000 to purchase, and an 8-GPU server system runs $300,000 to $500,000 before accounting for power, cooling, and maintenance. Lead times for data center GPUs currently stretch from 36 to 52 weeks. Memory prices are following the same trend: DDR5 RAM prices have surged over 400% since mid-2025, with a 32GB kit that cost around $80 in mid-2025 reaching approximately $432 by early 2026, and high-bandwidth memory suppliers have already sold out their entire production capacity through 2026, according to Newegg's 2026 memory market analysis.

For most engineering teams, individual ownership of AI infrastructure was never realistic. At today's prices and lead times, it's accessible only to hyperscalers willing to commit billions in advance. Microsoft, Google, Amazon, and Meta are each spending tens of billions on AI data centers and have multi-year supply agreements locked in. Everyone else rents. And renting means running your workloads on hardware pooled across many users simultaneously. This is shared infrastructure, and it is the default operating model for virtually every cloud-hosted AI coding assistant, model API, and hosted inference service on the market.

Here's what actually happens when you send a prompt to one of these tools:

Step

What happens

Who controls it

Request dispatch

Your prompt joins a queue alongside other users' requests

The provider

Hardware routing

The request gets assigned to a GPU server

Provider-managed, shared hardware

Memory load

Your code, prompt, and context load into GPU memory

On that shared hardware

Inference

The model processes your input

In memory, you have no visibility into

Response return

Output is returned to you

Hardware moves to the next request

For routine tasks on public or non-sensitive codebases, this trade-off is perfectly reasonable. The costs are manageable, the tools work well, and nothing about the workflow changes. The problem starts the moment the code you're sending is proprietary, regulated, or genuinely valuable, because at that point, the question of who else can access that hardware while your data is being processed becomes a matter of great concern.

What Happens to Your Code and Prompts on Shared Infrastructure

Most developers think of a prompt as something that feeds into a model and returns a response. The actual journey is longer than that, and at several points along the way, your code and context exist in plaintext on hardware you have no visibility into.

When a request reaches a shared inference server, it passes through three distinct stages before a response is generated:

Tokenization: Before the model sees anything, your prompt gets broken into tokens. This happens in CPU or GPU memory on the provider's hardware. Your raw text exists in plaintext at this stage, accessible to the host operating system and any monitoring tools running on that server.

Inference: The model processes your tokenized input inside GPU memory. This is where the actual computation happens, and it's also where your data is most exposed. On shared infrastructure, the same physical GPU that just processed another customer's request is now processing yours. The host OS and hypervisor sit above this execution layer and can, technically, access what's in memory.

Output generation: The model assembles a response token-by-token in memory before returning it to you. Until that response is serialized and sent back over the wire, it lives in the same shared memory space.

Here's what that looks like across a typical shared inference pipeline:

Stage

What's in memory

Who can technically access it

Tokenization

Your raw prompt, code, context

Host OS, provider monitoring tools

Inference

Tokenized input, intermediate model states

Host OS, hypervisor, provider tooling

Output generation

Assembled the response before transmission

Host OS, hypervisor, provider tooling

Logging

Request metadata, sometimes full payloads

Provider observability systems

Many providers enable request logging by default for debugging, quality monitoring, and reliability purposes. Even when explicit logging is disabled, observability tools often capture partial payloads or request metadata. A failed request can end up in a retry queue. An error trace can include the prompt that triggered it. None of this is malicious; it's standard operating procedure for running production infrastructure at scale. But standard operating procedure means your code and prompts are passing through systems designed to capture and retain information, and you have no way to verify what gets captured and what doesn't.

For a developer sending a public-facing feature or a learning exercise, this is background noise. For a developer sending proprietary business logic, a KYC verification flow, a pricing algorithm, or patient data as context, the exposure surface is real, and the only thing standing between your data and the infrastructure provider is a terms-of-service agreement.

Why This Risk Stays Invisible Until Something Goes Wrong

Shared infrastructure risk doesn't announce itself. The tools work exactly as advertised, fast completions, accurate suggestions, and smooth integrations. Nothing in the developer experience signals that anything unusual is happening with the data underneath. The risk is architectural, baked into how the infrastructure operates, and completely invisible at the surface level where developers interact with it.

Four specific vectors make this exposure concrete:

Provider logging and observability. To keep production systems running reliably, providers instrument their infrastructure heavily. Request queues, error traces, latency monitors, and quality pipelines all touch the data flowing through the system. Some providers explicitly log prompts and responses. Others disable logging by default but retain the capability to enable it. In either case, the infrastructure is designed to be observable, and your code and prompts are part of what flows through it.

Legal compulsion. Where your data is physically stored matters less than which legal jurisdiction governs the provider. The US CLOUD Act allows US authorities to compel disclosure of data held by US-headquartered companies regardless of where those servers are physically located. A provider can host infrastructure in Frankfurt, Singapore, or anywhere else, and still be legally required to hand over session data if compelled by a US court or government order. For teams operating under GDPR, HIPAA, or financial services regulations, this creates a direct conflict that geographic server selection alone cannot resolve.

Infrastructure compromise. A breach of a shared inference provider doesn't just expose stored databases; it exposes the full request history flowing through the system at the time of compromise, plus whatever has been retained in logs, retry queues, and observability systems. The attack surface isn't just the data at rest. It's everything that passed through the infrastructure in plaintext during processing.

Co-tenant side channels. On shared GPU hardware, multiple tenants' workloads run on the same physical chips. Research has demonstrated that side-channel attacks on shared GPU memory are theoretically possible, exploiting timing differences, memory access patterns, or residual state left by a previous tenant's workload. This is a harder attack to execute than logging or legal compulsion, but it represents a class of risk that exists structurally on any shared hardware and cannot be closed by software controls alone.

The common thread across all four is that none of them requires the provider to behave badly. Logging is standard practice. Legal compulsion operates outside the provider's control. Breaches happen to well-intentioned companies. Side channels are a property of shared hardware, not a policy choice. The exposure exists regardless of how trustworthy the provider is, which is precisely what makes it difficult to reason about and easy to dismiss until it becomes a problem.

Why Policy Promises Are Not Technical Guarantees

Every major AI development tool on the market makes security commitments. Terms of service promise data isolation. Compliance certifications attest to security processes. Enterprise agreements include "do not train" clauses. For many use cases, these assurances are sufficient. For teams sending proprietary or regulated code through these tools, the gap between what a policy promises and what it can actually guarantee is worth understanding precisely.

A policy is a statement about intended behavior. When a provider says your data is isolated, they mean their systems are configured to keep it that way, and their employees are contractually bound not to access it. That commitment can be audited; third parties can verify that the processes described in the policy exist and are followed. What a policy cannot do is prevent a misconfiguration, override a legal compulsion order, or stop a breach from exposing data that was accessible during processing. The data was reachable. The policy just said it wouldn't be touched.

"Do not train" clauses are the most commonly cited protection, and the most commonly misunderstood. Even where they hold, they only address one specific risk, whether your data is used to train future models. They say nothing about whether your prompts are logged, retained for debugging, accessible to provider employees, or subject to legal disclosure. A provider can honor a "do not train" clause perfectly and still have your code sitting in a request log for 90 days.

Compliance certifications like SOC 2 work similarly. SOC 2 attests that security controls were in place and followed at the time of the audit. It verifies processes, not individual execution events. A SOC 2 report issued six months ago says nothing about what happened to your specific request last Tuesday, whether it was logged, which nodes processed it, or what state the infrastructure was in at that moment.

The practical difference between these assurances and cryptographic proof looks like this:

Security claim

What it actually guarantees

Who can verify it

"We don't access your data"

A contractual commitment

Third-party auditors reviewing processes

"We are SOC 2 compliant"

Controls were in place at audit time

Auditors at the point of certification

"Do not train" clause

Legal protection in jurisdictions that recognize it

Lawyers, not engineers

"We are GDPR compliant"

Processes align with regulation at the time of review

Regulators and auditors periodically

TEE cryptographic attestation

The workload ran in an isolated, verified environment

Anyone with the attestation record, at any time

The last row is structurally different from everything above it. Cryptographic attestation isn't a promise about behavior; it's mathematical evidence that a specific workload ran inside a specific, verified execution environment at a specific point in time. That evidence is independently verifiable by anyone with access to the attestation record. It doesn't depend on trusting the provider's processes, their employees' behavior, or their legal obligations to a foreign government.

For teams where the question "can you prove what happened to our code during inference?" needs a technical answer rather than a contractual one, the distinction between these two categories is where the conversation about AI development tooling actually starts.

What Hardware-Level Isolation Actually Does

To understand why hardware-level isolation closes the gap that policies can't, it helps to understand what actually happens inside a CPU and GPU during inference, and where the boundary between "your execution" and "everyone else's execution" normally sits.

On standard shared cloud infrastructure, that boundary is enforced by software. The hypervisor, the software layer that manages virtual machines on shared hardware, separates tenants from each other and from the host. Software-level isolation works well under normal conditions. The problem is that the host OS and hypervisor sit above the execution layer, which means they can, in principle, access the memory of any workload running beneath them. A sufficiently privileged process, a compromised admin account, or a misconfigured monitoring tool can cross that boundary. The isolation is real, but it's contingent on the software stack behaving correctly.

Trusted Execution Environments move the boundary from software to silicon. A TEE is a hardware-enforced isolation mechanism built into the CPU itself. Memory inside a TEE is encrypted at the hardware level; the encryption keys are managed by the CPU, not by the operating system or the hypervisor above it. Even the provider's own infrastructure cannot read what's happening inside an active TEE. The host can schedule the workload and allocate resources, but it cannot inspect the memory where the computation is actually happening.

Intel TDX (Trust Domain Extensions) is the specific TEE technology relevant here. TDX creates encrypted virtual machine domains, Trust Domains, where code executes without exposure to the host OS or other tenants on the same hardware. Each Trust Domain has its own encrypted memory region. Even if the host is compromised, the data inside the Trust Domain remains encrypted and inaccessible.

The trust model this creates is meaningfully different from software isolation:

Software isolation (standard cloud)

Hardware isolation (TEE)

Who enforces the boundary

Hypervisor and OS

The CPU itself

Can the host read workload memory

Yes, with sufficient privilege

No, memory is hardware-encrypted

Can a compromised admin access data

Yes

No

Can a misconfigured tool expose data

Yes

No

Basis of trust

Provider's software stack

CPU manufacturer's hardware design

NVIDIA GPU Attestation extends this model to GPU workloads specifically. For AI inference, the computation that matters most happens on the GPU, not the CPU. NVIDIA GPU Attestation verifies that a GPU workload is running in a trusted, unmodified environment, producing cryptographic evidence that can be validated independently. For model inference requests sent through ORGN, both Intel TDX and NVIDIA GPU Attestation are verified per request, meaning each model call produces a cryptographic record proving it ran in a verified hardware environment.

That record, the attestation, is what makes hardware-level isolation verifiable rather than just claimed. Attestation works like this:

• Before a workload runs, the TEE generates a cryptographic measurement of its own state, the exact code loaded, the hardware configuration, and the runtime environment
• That measurement is signed by the hardware itself using keys that only the CPU or GPU holds
• The signed record can be verified by anyone with access to it, independently of the provider
• If the environment has been tampered with or doesn't match the expected configuration, the attestation fails, and the connection is blocked

For a developer or a compliance team asking, "Can you prove that our code was processed in an isolated environment and not exposed to the provider's infrastructure?", an attestation record is a technical answer. Every other security commitment on that list in the previous section is either legal or procedural.

How ORGN Eliminates Shared Infrastructure Risk

Most AI development tools are built on top of shared infrastructure and then add security controls on top. ORGN is built the other way around; the isolation is the foundation, and the development environment is built on top of it.

Every workspace runs inside a TDX Sandbox by default. From the moment a project is created and deployed in ORGN, its workspace runs inside a Confidential Sandbox backed by an Intel TDX-encrypted CPU and memory. This isn't an enterprise add-on or a configuration option; it's the default for every workspace, visible in the status bar of every active session. The host infrastructure cannot read what's happening inside your workspace, regardless of what model you're using or what code you're working on.

Model requests sent to ORGN models are backed by both Intel TDX and NVIDIA GPU Attestation. ORGN's unified AI gateway that routes requests to models running inside Trusted Execution Environments when confidentiality is required. For every request sent to an ORGN confidential model, both Intel TDX and NVIDIA GPU Attestation are verified, producing a cryptographic attestation record that's visible and retrievable in the ORGN Scanner. Teams can inspect exactly what ran, on what hardware, and when, per request, not per audit cycle.

Here's what that looks like in practice for a fintech team refactoring a KYC verification module:

• The agent reads the relevant files inside a TDX-encrypted workspace
• Changes are proposed across multiple services and run against the test suite in an isolated Trial
• Every model inference request sent to ORGN Gateway generates an attestation record
• Nothing merges until a developer approves it
• When the compliance team asks how the code was produced and where it was processed, the attestation record in the ORGN Scanner provides a cryptographic answer

The same workflow applies to an individual developer building a proprietary ML pipeline, the hardware-level isolation doesn't require a procurement process or an enterprise agreement. It's available from the first $20 of credits.

Data persistence is user-controlled throughout. Nothing is retained unless the user actively chooses it. Worktree data follows a defined lifecycle before teardown, and users can trigger immediate teardown at any time by archiving and deleting their worktree. Throughout that lifecycle, all data sits inside a TDX-encrypted sandbox, meaning even retained data is encrypted at the hardware level and inaccessible to anyone other than the user. For ORGN confidential models specifically, no prompts, code, or outputs are stored or used to train AI models at any point, except for operational metadata such as token counts needed for billing.

Secrets stay scoped to the workspace. Environment variables and API keys are injected into the sandbox at runtime and never committed to GitHub. They're managed through the Secrets tab in project settings, scoped to the workspace, and don't persist outside the execution environment.

The workflow itself is structured around isolation. ORGN's Projects → Tasks → Trials → Worktrees hierarchy isn't just an organizational structure; each Trial runs in its own isolated sandbox. An agent working on a task operates within a scoped environment with its own memory, execution context, and teardown lifecycle. Parallel agentic sessions can run simultaneously without sharing state or bleeding context between workstreams.

Here's how ORGN's architecture compares to standard shared inference infrastructure:

Standard shared inference

ORGN

Workspace isolation

Software-level, provider-managed

Intel TDX hardware encryption, default for all workspaces

Model inference isolation

Shared GPU, no attestation

Intel TDX + NVIDIA GPU Attestation per ORGN request

Data persistence

Provider-controlled, varies by policy

User-controlled, immediate teardown available

Training on your data

Governed by "do not train" clause

No prompts, code, or outputs stored for ORGN confidential models

Proof of execution

None, policy-based assurances

Cryptographic attestation record per request, retrievable in the ORGN Scanner

Secrets management

Varies by tool

Scoped to workspace, never committed to GitHub

Pricing is credit-based and pay-as-you-go. Self Serve starts from $20 in prepaid credits, no subscription, and credits never expire. Enterprise pricing is available on request. Get started at orgn.com.

The Developers and Teams for Whom Shared Infrastructure Risk Is Not Theoretical

Shared infrastructure risk comes down to one question: how sensitive is the code and context being sent through these tools, and what would the consequences be if a third party could access it?

Developers building proprietary algorithms, models, or core business logic. A recommendation engine, a pricing model, and a fraud detection system represent genuine competitive IP. For early-stage companies where the algorithm is the product, that exposure is meaningful regardless of whether anything bad actually happens.

Regulated industries. Fintech, healthcare, defense, and legal teams operate under frameworks, GDPR, HIPAA, and financial services regulations, that don't distinguish between intentional exposure and infrastructure that was technically accessible. The question an auditor asks isn't whether anything went wrong. It's whether the technical controls were in place to prevent it.

AI-native startups in active development. The phase when a product's core architecture is being built is also when the most sensitive implementation details are flowing through AI tools. Early architectural decisions, novel approaches, unreleased features, all of this passes through the prompt. For startups where the technical differentiation is the moat, infrastructure choices during this period matter more than most teams realize.

Teams working under client confidentiality obligations. Consulting firms and contractors building systems for regulated clients often inherit those clients' compliance requirements. Sending client code through a shared inference provider can violate contractual confidentiality obligations even when it doesn't trigger a direct regulatory breach.

A useful framing: think of the prompt you're about to send as a document handed to an external contractor operating on their own systems, outside your direct control. If sending that document externally would require a legal review or an NDA, the same reasoning applies here.

Conclusion: The Infrastructure Underneath Your AI Tools Is a Security Decision

Shared infrastructure exists for good reasons; the economics of GPU compute make it the only viable model for most teams, and for much of development work, it's a perfectly reasonable trade-off. The risk becomes real the moment the code flowing through these tools is proprietary, regulated, or competitively sensitive. At that point, the gap between policy-based assurances and hardware-level proof stops being a theoretical concern and starts being an audit finding, a compliance gap, or an IP exposure that's difficult to quantify after the fact.

For teams that need to close that gap, ORGN is worth exploring. Every workspace runs inside an Intel TDX-encrypted sandbox by default, model requests sent to ORGN models are backed by both Intel TDX and NVIDIA GPU Attestation per request, and data persistence is user-controlled throughout. Self Serve starts from $20 in prepaid credits with no subscription required. If your team is working through broader questions around secure AI development, verifiable execution, or compliance in regulated industries, the articles on confidential computing for AI workloads and data sovereignty in AI development on the ORGN blog cover the technical foundations in depth. Get started at orgn.com.

FAQs

What is shared infrastructure in AI development tools, and why is it a security risk?

Shared infrastructure means your code, prompts, and context are processed on GPU hardware that is simultaneously serving other users. At every stage of inference, tokenization, model execution, and output generation, your data exists in plaintext in memory on hardware managed by the provider. The security risk is that the host OS, hypervisor, and provider observability tools sit above the execution layer and, in theory, can access memory contents. Software-level tenant isolation reduces this risk but cannot eliminate it the way hardware-level isolation does.

How is a TDX Sandbox different from a standard cloud VM or container?

A standard cloud VM or container relies on the hypervisor, a software layer, to separate tenants from each other and from the host. A TDX Sandbox enforces that boundary at the CPU level. Memory inside a TDX Trust Domain is encrypted by the hardware itself, using keys managed by the CPU rather than the operating system. The host can schedule and allocate resources, but cannot read the contents of memory inside the sandbox. Even a compromised administrator account or misconfigured monitoring tool cannot cross that boundary.

What does NVIDIA GPU Attestation verify for ORGN model requests, and how is it different from Intel TDX?

Intel TDX provides CPU-level isolation for the workspace runtime, the environment where your code, agents, and project context live. NVIDIA GPU Attestation specifically covers the GPU workload during model inference. For model requests sent to ORGN confidential models, NVIDIA GPU Attestation verifies that the inference ran on a genuine, unmodified GPU inside a trusted execution environment, producing a cryptographic record of that fact. Both attestations are verified per request for confidential models and are visible in the ORGN Scanner, meaning every individual TEE model call produces an independently verifiable proof of execution. ZDR models carry Zero Data Retention guarantees enforced by provider policy but do not run inside TEEs and do not generate attestation.

How does ORGN handle data persistence for confidential model requests, and what does that mean for teams with compliance requirements?

For model requests sent to ORGN confidential models, no prompts, code, or outputs are stored or used to train AI models at any point. The only data retained is operational metadata, token counts, and billing information. This applies at the infrastructure level, not as a configurable policy, meaning there is no logging toggle that an administrator could accidentally leave enabled and no stored request history subject to legal compulsion. For teams under GDPR, HIPAA, or financial services regulations, this removes the attack surface entirely for ORGN-routed inference rather than reducing it.

What is cryptographic attestation in the context of AI inference, and why does it matter for regulated industries?

Cryptographic attestation is a hardware-generated proof that a specific workload ran inside a verified, unmodified execution environment at a specific point in time. Before execution, the TEE generates a measurement of its own state, the loaded code, hardware configuration, and runtime environment, and signs it using keys only the hardware holds. That signed record can be verified by anyone with access to it, independently of the provider. For regulated industries, this matters because it answers the auditor's question, "Can you prove what happened to our data during inference?", with mathematical evidence rather than with a compliance certification or a terms-of-service commitment.