Amazon Web Services launched its Continuum security agent for code vulnerabilities on Thursday, claiming the tool passed 819 of 920 tasks on the CyberGym-E2E benchmark for an 89.0% end-to-end success rate. The service locates and validates bugs in repositories, verifies them against AWS environments, and submits pull requests with suggested repairs. AWS published the 89% figure on October 5, more than two weeks before making Continuum generally available, marking what CEO Matt Garman called a higher score than any other AI agent on the test.
The CyberGym-E2E benchmark gives an agent 90 minutes to locate and repair a vulnerability in one of 139 open-source projects without receiving a bug description, proof of concept, crash log, or existing patch. Continuum identified and reproduced a bug in 92.5% of tasks, stopped the crash in 89.6%, and kept the project's tests passing in 89.0%. The benchmark's fourth stage checks whether the patch also resolves the historical vulnerability the test selected, and Continuum's patch fixed that specific historical bug in 37.8% of tasks, compared to a previous public high of 26.2%. All tasks involve memory-safety bugs in C or C++, leaving out many other languages and vulnerability types. When given more time beyond the 90-minute limit, AWS says the success rate climbs to 93.7%.
According to AWS, Continuum followed the benchmark's rules, including limits on internet access, and the company verified that the agent wasn't simply retrieving fixes already public. The company explains that a repository can contain multiple vulnerabilities, and repairing a different one doesn't render the work useless, which accounts for the difference between the two scores. Continuum completed 89% of tasks by finding and patching a real vulnerability without breaking existing tests, but only 37.8% of those patches addressed the historical bug the benchmark was checking for, and the agent wasn't told which bug that was.
The gap between Continuum's 89% task completion rate and its 37.8% historical-bug fix rate reveals a key limitation: the agent excels at discovering and repairing *a* vulnerability but doesn't always fix the *specific* vulnerability a team might need addressed. AWS ran the tests itself, and while the benchmark and testing tools are public, these results come from AWS alone. What remains unknown is how well Continuum performs when a team directs it to fix a particular vulnerability rather than letting it hunt freely. The benchmark's focus on C and C++ memory-safety bugs also means the 89% figure doesn't reflect performance across other languages or vulnerability categories, a scope limitation AWS acknowledges.
The results suggest Continuum can substantially reduce the manual effort required to patch codebases, but teams will likely need to verify that the agent's chosen fix aligns with their priorities rather than assuming it tackled the most urgent flaw. The 37.8% rate of fixing the benchmark's target bug, while higher than previous public benchmarks, still means nearly two-thirds of successful patches addressed a different vulnerability than the one the test was designed to measure. Organizations considering Continuum will need to weigh the speed of automated patching against the uncertainty of which specific vulnerabilities get resolved first, especially since the scope of testing remains narrow and vendor-validated. For security teams accustomed to prioritizing known critical exposures, delegating triage to an agent introduces a layer of unpredictability that may demand new workflows and verification steps.