AI coding agents continue to struggle with large-scale refactoring work, with the highest-performing model reaching just a 41.2% success rate on a new benchmark designed to test these capabilities, according to research from Shanghai Jiao Tong University, Peking University, Douyin Group, and partner institutions. The benchmark, called SWE-Bench ProMax, focuses specifically on refactoring tasks — restructuring code without changing its behavior — across 170 real-world instances spanning seven programming languages. The researchers say existing benchmarks fail to capture this critical skill, allowing models to post impressive scores on simpler tasks while missing abilities developers actually need.

SWE-Bench ProMax draws from actual code commits in Python, Java, TypeScript, Go, C, C++, and Rust, with each instance passing through multiple review stages to fix problems that plague other tests. The creators rewrote issue descriptions for greater precision, manually checked test suites to eliminate overly narrow or broad tests, and filtered out tasks lacking sufficient complexity or cross-file scope. A recent audit the researchers cite found nearly 60% of unsolved instances in SWE-bench Verified contained flawed tests, and evaluation quality for AI coding agents is declining as frontier models can breeze through benchmarks when solutions leak into training data. The new benchmark aims to address these shortcomings by presenting what the authors call "a meaningful and unsaturated challenge" that current AI systems can't easily game.

The report explains why refactoring proves so difficult for AI agents, quoting Shane Warden, principal architect at ActiveState: "Strict refactoring demands zero tolerance for error, zero tolerance for behavior changes, and complete reversibility." Vojtěch Pavlík, senior director of technical strategy for core infrastructure at SUSE, adds that fundamental capabilities remain out of reach, telling The New Stack: "We don't have LLMs that would be able to read a large codebase and see it in its entirety and 'understand' it all at once." Warden says many engineers treat refactoring as a text-generation task, feeding entire codebases into large context windows and waiting for multi-file diffs, but he rejects the underlying assumption: "This approach relies on the premise that current LLMs express a deep understanding of large systems. I don't believe that premise. I believe that token proximity does not guarantee structural understanding."

The report identifies several technical barriers blocking better refactoring performance. Pavlík points to attention mechanisms as a core problem, explaining that highly optimized attention algorithms like Deepseek Sparse Attention are required for usable performance in very large models, but for changes in complex codebases that max out the context window, "this can easily result in missing important observations and generally getting confused." He also calls out time as a fundamental factor that trips up language models, citing race condition errors, loss of idempotency and atomicity, and incorrect retry handling — issues that emerge when parallel tasks execute in different orders. Models produce code that passes tests and works "on a sunny day" when users follow exact specifications, he says, but fails when a user clicks a button twice quickly, a connection times out, or two tasks finish simultaneously and overwrite each other's results.

To improve agent refactoring capabilities, the experts recommend multi-layered approaches that don't rely solely on language models generating raw code diffs. Pavlík calls for analyzing and mapping codebases to create detailed specifications, using retrieval-augmented generation to recall relevant code sections rather than depending purely on attention, bringing humans in for validation, and implementing comprehensive test suites before starting any refactoring work. Warden describes a more promising architecture where the language model identifies code smells, evaluates design trade-offs, and queries Language Server Protocols and code graph tooling to analyze explicit structures, dependencies, and data flows, then issues small, zero-entropy commands to execute changes and rolls them back if tests fail. Pavlík sees SWE-Bench ProMax as a signal that benchmarks are shifting from measuring how good models are overall to targeting specific tasks where they need to improve, making tests more complex and less learnable upfront so they better expose where agents actually struggle. Benchmarks can't perfectly measure real-world performance, but excluding difficult tasks like refactoring creates an overly optimistic picture of what agents can do, and this new test brings a dose of realism. The gap between marketing promises and engineering reality matters because organizations betting on AI agents need to know which tasks still require human developers, and which quality-control processes can't yet be safely automated.