Cloudflare has rolled out a beta Web Search API through its AI Gateway to solve a persistent problem: AI agents that blindly guess URLs and return error pages. Available via REST APIs and Cloudflare Workers, the service delivers live web results from three providers—Ceramic.ai, Exa, and Linkup—with unified logging, access controls, and billing. The API lets AI applications pull current web content, including recent news, API changes, and technical documentation, and feed those results as context to language models during inference.
The system provides a single interface for multiple search providers, integrates search billing directly with AI Gateway, and logs search requests alongside model inference. Applications and external services can tap into web search through a REST endpoint by supplying an AI Gateway authentication token and identifying a search provider in the request payload, which returns the search results. Cloudflare states it will flag providers that support Zero Data Retention and offers search at providers' list prices without extra markup. Users can bring their own provider keys.
Michelle Chen, group product manager at Cloudflare, along with Sam Else, senior director, and Gabriel Massadas, senior systems engineer, describe the issue the company aims to fix: "When you use an agent and it needs to fetch a live web page, the agent usually just guesses the URL of the page and then makes a tool call to curl it. This is why you'll sometimes see web fetches come back with a 404 Not Found." The authors add that the crawler used by the web search provider must comply with Cloudflare's publicly stated requirements for "Verified bots," and web search responses must include a link to the location of crawled content, rules they say are "net-positive for a fair Internet."
The search API addresses a workflow inefficiency where agents randomly guess URLs, but it also raises questions about data retention and competitive advantage. In a popular Hacker News thread, users debate search quality, data-retention restrictions, and whether a unified API delivers meaningful benefits over existing alternatives. Simon Willison raises concerns about storing and resyndicating results, noting that if an agent system can't store responses or offer a "share transcript" button, "that's a pretty significant limitation." Maximilian Goisser, CTO and co-founder of Erato Labs, observes that Cloudflare is approaching the status of trusted main cloud supplier for companies already using it, positioning itself as the platform through which the majority of cloud costs are billed so customers don't face a fresh procurement process.
Cloudflare plans to add native server tools to AI Gateway, starting with web search; until then, developers can integrate web search into their workflows using Cloudflare Workers. The service requires providers to follow verified-bot requirements and return source links, a move intended to let creators decide what they want to do with their data. By offering search at list prices without markup and supporting bring-your-own-provider keys, Cloudflare positions the API as a cost-neutral aggregation layer rather than a premium service. For organizations already embedded in Cloudflare's ecosystem, consolidating AI inference and web search billing under one roof may streamline operations, though transparency around data retention will likely shape adoption among developers building shareable agent systems.