When asked to build customer service agents entirely on their own, the best AI developer systems succeeded just 23.9% of the time, according to Hyper-τ-bench, a new benchmark released in early September by Sierra, the enterprise AI agent company co-founded by OpenAI board chairman Bret Taylor. The test measured how well AI systems could independently construct other AI agents — gathering business requirements, writing code, and producing working customer service systems — across simulated airline, retail, telecom, and banking scenarios. None of the six autonomous setups tested broke 25%, far below the 82.2% achieved when a human engineer with deep context worked alongside a frontier model.
Sierra researchers tested six combinations of AI model and coding environment, including Anthropic's Claude Opus 5 in Claude Code, OpenAI's GPT-5.6 Sol in Codex, and Moonshot AI's Kimi K3 in both Kimi Code and the open-source OpenCode. Claude Opus 5 in Claude Code led at 23.9%, narrowly ahead of GPT-5.6 Sol in Codex at 22%. Performance varied dramatically by industry: Claude Opus 5 reached 72.8% on retail tasks, 55.9% on airline, and 48.2% on telecom, but plummeted to just 5.9% on banking. GPT-5.6 Sol managed 9% on banking scenarios. Banking accounted for 35 of the benchmark's 53 construction tasks and contained 2,969 individual policy facts, with a single task sometimes depending on as many as 580 of them. Average build times ranged from 40 to 63 minutes, with token spending between 3.4 million and 9.1 million tokens per build.
The failures weren't uniquely machine-like mistakes, according to the researchers. "The failures mirror ones human agent developers see," the paper states. Developer agents stopped researching the business too soon, opening fewer than 80 of roughly 1,700 available files in banking scenarios and instead relying heavily on searches for documents that appeared relevant. They rarely asked questions when information was missing — such interactions accounted for just 0.3% of tool calls across recorded runs, even though some tasks required asking 20 to 25 questions to uncover requirements that couldn't be found in documents. On tasks where the expert-built reference scored between 95% and 100%, builds that asked no questions scored just 5%, rising to 15% after one question and 25% after two. The agents also showed little appetite for trying different technical approaches: 92% of builds used a "single LLM tool loop" design, and in one telecom experiment, giving the developer agent a single sentence suggesting a different architecture lifted its score from 31% to 67%.
Sierra researchers Ben Shi and Keshav Dhandhania argue that building an enterprise agent is often "less like implementing a spec, and more like doing research" — requirements are scattered across handbooks, support systems, spreadsheets, and the knowledge of frontline staff. The autonomous developer agents could write code and produce functioning systems, but they struggled with judgment-heavy parts of the job: gathering enough information, knowing when to ask questions, and exploring alternatives rather than settling quickly on an answer. The benchmark reveals a blind spot: because the system being built is itself an AI, the only way to know if a design works is to run it and observe what it says to real users, whom the developer never sees while building. Sierra's results suggest the developer agents often failed to compensate for that blind spot through enough testing and iteration, instead "shipping the first design that runs." AI is already taking on more of the work involved in building agents through tools like Sierra's Ghostwriter, Microsoft's Copilot Studio, and Salesforce's Agentforce Builder, but humans still provide business context, decide what good looks like, and check the result. The benchmark asks what happens when you remove much of that human guidance — and for now, its results suggest autonomous developer agents still struggle with some of the judgment-heavy parts of the job. The gap between autonomous performance and human-assisted builds may narrow as models improve, but the test exposes a persistent challenge: gathering scattered requirements and making architectural trade-offs without direct visibility into end-user experience remains difficult territory for systems working alone.