Large language models produce flawed vulnerability patches more than half the time when tackling complex security issues, according to a new study from 1Password. Researcher Keith Hoodlet disclosed that LLMs generate what the team calls "Fix-Like Artifacts with Embedded Defects" 53.9% of the time when sophisticated patches are needed. The internal evaluation found that AI-created fixes routinely miss wider issues like architectural intent, business needs, security ramifications, and future maintainability, even when the code syntax appears correct.

1Password tested 6,080 patches created by ChatGPT-5.5 and Claude Opus 4.8 across six recently disclosed vulnerabilities, including CVE-2026-31431, CVE-2026-34197, CVE-2026-8512, CVE-2026-45185, CVE-2026-22738, and Gemini CLI RCE. Only 26% of the patches fully eliminated the flaw without changing how the application worked. Another 20.1% resolved the vulnerability but altered application behavior in the process. Nearly half—49.3%—failed to close at least one exploitable attack route, while 2.3% fixed the original weakness but created a new one, and 2.2% both failed to address the issue and introduced an additional security problem.

The researchers discovered that passing predefined tests can mask deeper failures. More than one-third of patches that initially looked successful were labeled "fragile" because they merely blocked the proof-of-concept exploit shown during testing rather than fixing the underlying root cause. Hoodlet pointed to the SpringAI CVE patches as an example: both GPT and Claude models generated fixes that targeted specific characters from the input string in the proof-of-concept they received, leaving the fundamental issue intact. "If the guarded code were to become reachable again by using alternative inputs, it would lead to the old vulnerability resurfacing in the software," he noted.

1Password argues these failures stem from the contextual reasoning necessary to create production-ready security fixes. The company reached out to Anthropic, which reportedly advised keeping humans involved in the process. "Patch generation has outpaced patch verification, and the fix is to make verification execution-grounded rather than inspection-based, while keeping domain experts as the final reviewers at current model capabilities," the company was quoted as saying. 1Password also pushed back against the idea that AI-generated patches come without cost. While the typical patch-and-validation cycle ran approximately $2.11 using ChatGPT-5.5 and $2.81 using Claude Opus 4.8, Hoodlet contended that the genuine expense sits in confirming whether those patches are secure enough for production deployment. Human review remains the final security control, according to the report. Organizations betting on automated patching without expert validation may be trading short-term speed for persistent exposure, and the shift toward AI-assisted remediation will likely force security teams to rethink verification workflows rather than eliminate them.