Microsoft has launched a dedicated AI Gateway tier of Azure API Management in public preview, a gateway resource whose control plane organizes around models, MCP servers, and tools instead of traditional APIs. According to a report published by InfoQ, the gateway was designed as a standalone experience rather than an additional policy layer on existing infrastructure, reflecting the reality that teams now interface with multiple AI providers instead of relying on just one. The preview makes available Foundry-hosted models from OpenAI, Anthropic, and Mistral, plus models in AWS Bedrock, Google Vertex AI, and OpenAI directly.
The gateway unifies all OpenAI-compatible providers under a single endpoint path, routing requests based on an exact match of the model field, which means each published model requires a distinct name. The report notes that Anthropic is managed through a custom provider with Messages API passthrough. Policies appear as cards in the portal instead of XML and expressions, spanning token and request limits, quotas, Content Safety, and model fallback. A gateway provisions in roughly one minute with no scale units to configure. Telemetry exports as OpenTelemetry token metrics to Application Insights, Datadog, Grafana, and other customer-controlled destinations, and the resource operates within the customer's own subscription and Entra tenant. For tools, the gateway federates backends from three sources: a remote MCP server by URL, an OpenAPI specification, or a built-in connector covering over a thousand SaaS applications with no server to host. Each backend's operations convert into tools, and teams select per backend whether the gateway authenticates with none, an API key, OAuth 2.0, or a managed identity.
The intended operating model divides central control from team self-service, the report explains. A platform group connects approved models and tools and publishes them, while application teams use those assets in a test console and develop against them without routing every modification through the center, as the platform group maintains guardrails and the usage overview. Reactions from architects and platform engineers were broadly positive on the consolidation, though questions emerged about where the governance boundary falls. Paolo Perrone, who writes the AI Engineer newsletter on production AI systems, argued in a comment on the launch that the underrated element is positioning cost governance at the gateway: "Most teams bolt on rate limiting and spend tracking only after an incident," and centralizing it provides "one control plane instead of per-app patches." Adolph White Jr., an enterprise AI systems architect, raised the sharpest open question about what occurs when an agent run doesn't finish cleanly: if an agent produces useful work but the run ends without clean completion, is that output preserved for auditable review, or does the gateway fail over and retry? He framed the distinction as one between "governing AI traffic and governing the full lifecycle."
The report identifies a design decision that merits scrutiny before teams plan an onboarding model around it. A runtime access key is gateway-scoped, reaching every model and every tool published on that gateway. Microsoft's guidance recommends one key per application, but the blast radius of a leaked key spans the whole gateway instead of a single product, and teams using APIM subscriptions today to scope consumers to a set of APIs won't find that boundary here. The preview posture warrants careful reading too: availability is best effort with no SLA, and APIs, telemetry, limits, regions, and pricing can shift before general availability. Preview quotas cap models, tools, runtime keys, and throughput, with specific limits unpublished. Pricing will be announced later in the preview, which makes the cost-governance argument the least settled portion of the release. The open question the announcement doesn't answer is coexistence, and the confusion is already visible. Rajib Mahapatro, an AI engineer and architect, responded to the launch by reporting four months of production use, citing caching, content safety, logging, and token limits—those are the existing capabilities rather than a tier that entered preview days earlier, which is precisely the ambiguity at issue. The AI Gateway tier is available now in East US 2 and Sweden Central, free during preview, with labs covering how to publish a governed Foundry model and watch token rate limits in action. Organizations that platform teams can now standardize multi-cloud AI access through a single control plane, though the line between infrastructure governance and application logic remains contested territory where engineering culture will ultimately dictate which layer owns the decision rights.