Insights

On-Premise AI vs Azure OpenAI With Private Endpoints

A private endpoint takes Azure OpenAI traffic off the public internet. It does not move the model into your building, and the data terms stay the same. Here is where the line falls.

Share:
On-Premise AI vs Azure OpenAI With Private Endpoints

An Azure OpenAI private endpoint changes one thing: the road a request travels. Instead of reaching a public hostname over the internet, the request goes to a private IP address inside the company's Azure virtual network and crosses Microsoft's backbone to the model. The model still runs in a Microsoft data center, on Microsoft's hardware, under Microsoft's service terms. So the comparison of an Azure OpenAI private endpoint vs on-premise AI comes down to whether the risk being managed is the network path or the processing itself.

That distinction matters because "private endpoint" sounds like "private AI." It is a networking feature, and a good one. It is not a change in who operates the model.

What a private endpoint actually is

Microsoft's private endpoint overview defines it as a network interface that takes a private IP address from the customer's virtual network and connects privately to a service powered by Azure Private Link. In Microsoft's words, enabling one brings the service into the virtual network.

For Azure OpenAI, which sits under Foundry Tools in current documentation, the virtual network guide says traffic between clients and the resource "traverses the virtual network and a private link on the Microsoft Azure backbone network, which eliminates exposure from the public internet." Offices reach the virtual network over a site-to-site VPN or ExpressRoute private peering.

Three details trip people up:

  • A private endpoint does not close the public door by itself. The overview page says private endpoints provide a private address "but don't necessarily restrict public network access." The resource's networking setting has to be changed to selected networks and private endpoints, or to disabled, and Microsoft's guide warns that network rules have no effect until the default rule is set to deny.
  • DNS does the routing. The resource's hostname resolves to the private IP only from inside the virtual network (or from a DNS server set up to forward the privatelink zone). Resolved from anywhere else, the same name points to the public endpoint. A misconfigured office DNS server quietly sends traffic the public way.
  • Clients keep the same URL. Microsoft's guide says requests must use the resource's custom subdomain, not the internal privatelink hostname. Application code usually does not change; the network around it does.

Private Link is billed separately: an hourly charge per endpoint plus a charge per GB processed, according to Microsoft's Private Link pricing page, which also notes that partial hours are billed as full hours and that ordinary data transfer charges apply on top. In October 2026 that page showed the rates as blank placeholders, so current figures have to come from the Azure pricing calculator.

What it changes and what it leaves alone

Follow one prompt, a paralegal asking for a summary of a client contract, through both setups.

Stage Public endpoint Private endpoint
Laptop to Azure Over the internet to a public IP, TLS-encrypted Over VPN or ExpressRoute to a private IP in the virtual network
Who can reach the resource Anyone with a valid key or token, unless IP rules are set Only clients inside approved networks, once public access is off
Where the model runs Microsoft data center Microsoft data center (unchanged)
Real-time content filtering Yes Yes (unchanged)
Abuse monitoring and possible human review Per Microsoft's terms Per Microsoft's terms (unchanged)
Stored data from optional features In the resource's Azure geography In the resource's Azure geography (unchanged)

Only the first two rows move. Every row about what happens to the text after it arrives stays where it was. The private endpoint removes a stolen API key used from a coffee shop as an attack path, and it removes the internet as a transport. It does not remove Microsoft from the processing chain, which is the part a confidentiality rule usually cares about.

Processing location

Microsoft's data privacy page for models sold by Azure says prompts and responses are processed within the customer-specified geography, unless the deployment type is Global or DataZone. A Global deployment may process prompts in any geography where the model is deployed. A DataZone deployment may process them anywhere in the data zone: a resource in the United States can be processed anywhere in the United States, and a resource in an EU member state in that or any other EU member state.

Take a firm with its resource in one EU country that picks a DataZone deployment. Its prompts may be processed in any of the 27 member states, and the private endpoint in its home region does not narrow that list to 1. The endpoint decides how traffic enters Azure; the deployment type decides where the model runs. Data at rest, including the abuse monitoring store, stays in the customer-designated geography under both deployment types, per the same page.

Abuse monitoring and human review

By default, prompts and completions flagged by classifiers or found in a potentially abusive pattern can be sampled for review. Microsoft's abuse monitoring page says review is automated first, and authorized Microsoft employees may review content when automated review falls below confidence thresholds or is unavailable. For EEA deployments, those reviewers are located in the EEA.

Customers can apply for modified abuse monitoring, which removes the storage and human review step. The limited access page says this is currently available only to customers and partners managed by a Microsoft account team or under an eligible program. A 15-person firm buying Azure on a credit card should not assume it qualifies. Even after approval, automated review may still run on prompts as they arrive. The private endpoint has no effect on any of this.

The question of whether prompts train models is a separate one, covered in the guide to reading a provider's data-use terms. Microsoft states that prompts are not used to train foundation models without the customer's permission and are not available to OpenAI.

Operator access

Microsoft personnel run the service. Customer Lockbox lists OpenAI and Foundry Tools among supported services and lets a customer approve or deny an engineer's request to access customer data during support; an unanswered request expires after four days. It needs at least a Developer support plan. Its own exclusions matter here: emergency "break glass" events, security or abuse investigations, accidental exposure during troubleshooting, and external legal demands do not trigger a Lockbox request.

What on-premise changes instead

An on-premise model server reverses the table. The network path is the office LAN, the model runs on hardware the company owns, and there is no provider abuse monitoring, no human reviewer at a vendor and no provider support engineer. Whoever holds the admin password is the operator.

That moves the work as well as the risk. Patching, access control, logging and uptime become the company's job, and the model choices narrow to open-weight models that fit the hardware. The definition of private AI by where inference runs sorts these arrangements more precisely: Azure with a private endpoint is a well-governed cloud service, while a model on the company's own server is private in the strict sense.

When each one fits

The deciding question is what the obligation on the data actually says.

Azure OpenAI with a private endpoint fits when:

  • The concern is exposure on the public internet, stolen keys, or a security policy that forbids internet-facing endpoints for business systems.
  • The company already runs workloads in Azure, with a virtual network, VPN or ExpressRoute and staff who maintain private DNS zones.
  • A contract or regulation allows a cloud processor under a data processing agreement, and the person accountable for the data accepts Microsoft's abuse monitoring terms as written.
  • The work needs frontier-scale models that would not fit on a server the company can afford.

On-premise fits when:

  • A client contract, professional rule or regulation says the data may not be processed by a third party, or must stay at a named site.
  • Nobody may see the content outside the company, including a vendor's reviewers, and the company cannot obtain modified abuse monitoring.
  • The data is subject to a demand that it never leave a network the company physically controls, which no cloud networking feature satisfies.
  • Usage is steady enough that a fixed-cost server stays busy.

Running both is a common arrangement. A local model handles the restricted folders, and the Azure resource behind a private endpoint handles everything else. Splitting work this way needs a written rule for which data class goes where, described in the article on routing between local and cloud models.

A practical test before signing anything: write down the sentence "This text will be processed in Microsoft data centers in [geography], with automated abuse review and possible human review by authorized Microsoft employees unless modified monitoring is approved." If the person who owns the data's obligations can sign it, the private endpoint route is open. If the obligation is about where the data physically sits, the data sovereignty requirements point to hardware the company controls.

Frequently asked questions

Does a private endpoint make Azure OpenAI HIPAA or GDPR compliant?

No networking feature makes a service compliant on its own. Compliance depends on the contract terms, the deployment type, the configuration and how the company uses the service. A private endpoint is one technical control a risk assessment can point to, and this is general information, not legal advice.

Is a service endpoint the same as a private endpoint?

No. A service endpoint lets traffic from an approved subnet reach the resource's public endpoint over an optimized Azure path, and the resource then filters by subnet. A private endpoint gives the resource its own private IP inside the virtual network. Microsoft's guide notes both can be used in the same subnet for different resources.

Can an office reach a private endpoint without building an Azure network first?

Not directly. The private IP lives in an Azure virtual network, so office machines need a site-to-site VPN or ExpressRoute into that network, plus DNS that resolves the resource name to the private address. Without both, office traffic falls back to the public hostname or fails once public access is disabled.

Does a customer-managed key stop Microsoft from processing the prompt?

No. Microsoft's data privacy page says customer-managed keys can encrypt data stored at rest by features such as files, batch jobs and stored conversations. The model still has to read the prompt in plain form to answer it, so the key protects storage, not processing.

Ready to Modernize Your Business?

Get your AI automation roadmap in minutes, not months.

Analyze Your Workflows →