AWS has launched aws-bench, an open-source tool designed to measure how well AI agents handle actual AWS tasks like diagnosing configuration errors, setting up infrastructure, and managing live cloud systems. The benchmark differs from conventional static testing methods by deploying scenarios in temporary, real AWS accounts rather than simulated environments. According to the company, this approach delivers a more precise assessment of how agents perform the work that developers and teams routinely do on AWS.
Each benchmark creates a scenario in isolated AWS accounts, which includes a collection of AWS resources built using CDK stacks. After the scenario is ready, the agent being tested executes a task inside a sandboxed container with limited credentials. When finished, an automated verifier scores the outcome, either through an LLM judge or a programmatic check against the live AWS state. The benchmark ships with datasets containing both fundamental and complex tasks spanning observability, compute and data, databases and storage, multi-region EC2, serverless, streaming and IoT, reference architectures, and troubleshooting across multiple services. While the release aims at AI researchers and model providers, engineering teams can use the benchmark and expand the scenarios and tasks to match their specific needs.
The project builds on Harbor, an open-source framework for testing AI agents, adding features like AWS resource setup, scenarios, and verifiers. Built-in adapters work with several widely available agents and models, including Claude Code, Codex, Kiro CLI, and Mini-SWE-Agent, plus any agent that Harbor already supports, such as Gemini CLI and OpenCode. Running the benchmark requires more than installing it locally—users need credentials with access to their organization's management account and permission to control member accounts and organizational units. The setup currently operates only in the us-east-1 region and establishes persistent resources that may generate costs even when idle. AWS has not released any baseline results or a standardized leaderboard, with both items listed on the future roadmap. The benchmark and datasets are available on GitHub under the Apache-2.0 license.
The launch arrives during rising doubts about the reliability of agent evaluation benchmarks. Researchers from the Center for Responsible, Decentralized Intelligence at UC Berkeley published findings showing they built an agent that achieved near-perfect scores on prominent benchmarks like Terminal-Bench and SWE-Bench without completing a single task. The researchers stated: "These are not isolated incidents. They are symptoms of a systemic problem: the benchmarks we rely on to measure AI capability are themselves vulnerable to the very capabilities they claim to measure." As usage expands, the benchmark's ability to resist gaming will face close examination—most tasks in aws-bench use an LLM judge for evaluation, and AWS itself notes that residual state can cause unintended passes or random failures. Organizations deploying AI agents for cloud operations will need to weigh whether real-world testing environments justify the added complexity and cost, particularly when the industry lacks consensus on what constitutes a manipulation-proof evaluation standard. The broader challenge isn't just measuring agent performance accurately today, but ensuring that the metrics themselves remain meaningful as the agents they're testing grow more sophisticated at exploiting evaluation loopholes.