Even the most advanced AI models fail to resolve nearly four out of every ten complex IT support tickets, according to new benchmarks released by Shield Technology Partners. The evaluation raises questions about how managed service providers should balance automation with human supervision as they expand AI use in their operations. The findings show that technical capability alone doesn't guarantee successful service delivery, particularly when AI agents overlook customer instructions or established procedures.
The evaluation tested leading AI models against 52 challenging support tickets that Shield's AI Engineer had previously failed to resolve correctly. GPT-6 Astra configured with extra-high reasoning achieved the highest success rate at 61.5%, passing 32 of the 52 tickets, while Claude Opus 5.5 max and Astra on high reasoning effort each passed 59.6% of cases, resolving 31 tickets. Claude Opus 5.5 high passed 53.8% of cases, averaging 13 minutes per ticket at a cost of $3.30, while Gemini 3.8 Flash high passed 36.5% of cases, taking 24 minutes per ticket at a cost of $1.60. Raising Astra's reasoning effort from low to extra high resulted in two additional successful resolutions but increased processing time from under 10 minutes to nearly 20 minutes and cost from $5.10 to $7.60. DeepSeek V4.1 Flash low passed 25 tickets at a cost of $0.13. Shield noted that these figures don't include human verification, corrections, or follow-through, which can add to the overall expense of using lower-performing models.
The report finds that AI capabilities can vary significantly even within familiar IT workflows, with agents sometimes completing complex technical tasks while missing specific customer requirements or documented procedures. One example describes how a customer requested a phone call about a slow computer, failed updates, and printing issues, but Claude Opus 5.5 attempted to schedule remote maintenance through email instead, citing an earlier ticket as justification. According to Shield, this case demonstrates that evaluating AI performance requires more than determining whether an agent can complete a technical task—success also depends on whether the work meets customer requirements and follows established procedures. The organization said the findings are already helping frontier AI labs identify weaknesses in IT workflows, with researchers indicating they plan to use the evaluations to guide model improvements.
For managed service providers, the findings suggest that selecting AI models based on processing costs alone could create operational challenges. Lower-cost models may offer savings on individual tickets, but those advantages can diminish when technicians must review, correct, or repeat unsuccessful work. Shield notes that providers evaluating AI service desk tools will need to consider total resolution costs, escalation rates, and customer satisfaction alongside model performance. The organization is working with its partner companies to explore how IT service providers may need to reorganize as AI assumes more work, with one provider shifting some helpdesk capacity toward account management, projects, and managed AI services, while another restructures workflows to create a larger role for escalation engineers and introduce team leaders responsible for overseeing AI engineers. Jon Katz, COO of partner company LISS Technologies, said technicians are moving "from owning individual tickets to owning the process of a ticket," helping establish ideal outcomes, document which tasks can be delegated, and determine when human intervention is necessary. The transition could create opportunities to redirect technical capacity toward higher-value services without necessarily expanding headcount, the report concludes. Providers will need to weigh whether upfront automation savings justify the hidden costs of quality control and customer friction. The shift may ultimately favor firms that invest in oversight infrastructure rather than those chasing the lowest per-ticket processing fee.