A new framework published on The New Stack challenges the widespread industry belief that coding agents cannot be meaningfully assessed. The piece, written in response to claims from software factory providers that evaluation is impossible, argues that while coding agents are difficult to measure, they should be held to the same standards as traditional software systems. The framework lays out a multilayered approach that goes far beyond simple pass-fail grading.

The framework distinguishes between a coding agent and the model that powers it, noting that an agent comprises a model, a harness, tools, repository context, instructions, permissions, an execution environment, and a feedback loop. Altering any single component can substantially change outcomes, meaning that benchmark scores often reflect a specific model-agent-environment setup rather than the underlying model alone. The author proposes grading behavior instead of comparing output to reference patches, using executable contracts such as whether code builds, whether existing tests still pass, whether hidden tests for requested behavior pass, whether public APIs remain compatible, and whether performance limits are respected. The framework identifies six evaluation layers: outcome, change quality, trajectory, human intervention required, economics, and production impact, arguing that no single metric can capture all dimensions.

According to the framework, non-determinism should be treated as a statistical problem rather than evidence that evaluation is impossible. The author recommends running the same task multiple times with controlled starting conditions and reporting success distribution instead of showcasing the best demonstration. For production use, the framework states that teams should ask "how often can it solve this class of task, within our budget, without introducing an unacceptable failure?" The piece notes that ambiguous tasks with incomplete requirements should include interaction with a controlled user or product-owner simulator, allowing agents to identify unclear specifications and ask necessary questions. The author writes that "claiming that coding agents cannot be evaluated gives providers the benefits of an engineering product without the burden of engineering evidence."

The framework explains that executable contracts allow multiple valid implementations while maintaining clear definitions of acceptable behavior, and are much harder to manipulate than prose descriptions of what an agent believes it accomplished. While some qualities like maintainability and architectural fit require human judgment, the author points out that human-written software has never stopped being testable simply because code review involves judgment. An agent can make new tests pass by weakening existing assertions, hard-coding expected values, or producing enormous rewrites that happen to work today but would never pass human review. The framework references emerging research benchmarks like ICAE-Bench, which evaluates agents as interactive project builders, and Dialogue SWE-Bench, which isolates dialogue capability from raw coding performance, showing that ambiguity changes evaluation shape rather than making it impossible.

The framework concludes that the standard for evaluation should be usefulness rather than perfection, asking whether assessment helps teams make better decisions than relying on demos, anecdotes, or provider claims alone. Production will always contain surprises, and no private test suite will represent every future task, but reproducible evidence remains superior to unverifiable assertions. The author emphasizes that coding agents do more than code-completion models—they explore, plan, edit, execute, recover, and sometimes negotiate intent—and evaluation systems must observe that entire loop. The bottom line: difficulty does not equal impossibility when it comes to measuring agent performance. For organizations weighing substantial investments in agentic tooling, the choice between rigorous multi-dimensional testing and faith-based procurement may determine whether automation becomes a competitive advantage or an expensive distraction. The debate over evaluability is ultimately a debate over accountability in a market segment where vendor promises have consistently outpaced verifiable outcomes.