In February 2025, the FTC finalized a $193,000 settlement against DoNotPay, a platform that marketed itself as the world's first robot lawyer, after the agency found the company never tested its output against advice a licensed attorney would produce, according to a report published by ZDNET. The settlement highlights a growing accountability gap that's pushing enterprise software vendors to build human-in-the-loop oversight directly into AI systems. Four experts interviewed for the report describe how these safeguards work, where they fall short, and what separates genuine oversight from corporate window dressing.
Human-in-the-loop, or HITL, refers to design patterns where autonomous AI agents route complex decisions through human review before executing tasks or generating responses. The approach is gaining traction in sectors like healthcare operations, regulatory compliance, high-value financial transactions, and legal decision-making, where mistakes carry steep consequences. However, the definition remains broad enough that companies use the term fluidly for widely different systems and architectures. A vendor performing periodic security audits can claim human oversight even when no person reviews crucial tasks in real time, the report notes. Akash Thakur, an SRE architect and AI reliability engineer based in Canada, explained that the real issue isn't the model itself but how systems respond when AI is uncertain or wrong, treating model failures as worst-case scenarios rather than building real-time audits that catch mistakes before regulators or customers discover them.
Most HITL systems rely on the model's own confidence score to decide when a response needs human review, but Daniel Gamber, CEO of AI document processing platform Cambrion, said confidence scores measure the wrong thing entirely. "A confidence score tells you the machine could read the text," he explained. "It tells you nothing about whether the number is actually right." Thakur echoed the concern, noting that model confidence isn't the same as correctness, allowing confidently wrong outputs to sail past review thresholds without triggering any red flag. Even when systems correctly flag responses, the routing process deciding how to address the flag is often too binary, according to Asim Husain, co-founder of Alterion and former VP of engineering at Google. He outlined four types of error responses HITL systems should plan for: notify someone and let it through, mask the sensitive part and proceed, hold for approval, or quarantine and kill the session outright. The ideal response depends on an organization's risk tolerance, and the system should know who to escalate to—a destructive database mutation should land with the platform or security team that owns that system, while a financial transaction above a threshold routes to whoever owns transaction controls, Husain added.
The EU Artificial Intelligence Act, the most well-developed piece of modern legislation in its category, makes demonstrable oversight a core requirement for compliance, the report states. Husain noted that "demonstrable" carries significant weight, meaning an organization only clears compliance requirements if it can prove in court or before a regulatory agency that it implemented real oversight with a clear audit trail of physical evidence. That requires training human reviewers to challenge AI output when it could be wrong while maintaining detailed records of the review process deciding whether output needs alteration or is fit for submission. Yet even solid review protocols can be undermined by bias inherited during model training—a 2024 Government Accountability Office report found that the IRS's automated audit-selection system audited Black taxpayers three to five times more often than non-Black taxpayers, even with no direct race input field, likely due to aggressive scrutiny of earned income tax credits claimed by low- to middle-income families and self-filing practices, according to Stanford researchers. Successful implementations like Nominal, an automated accounts payable platform, make human approval mandatory for transaction clearance, with AI providing context for human reviewers instead of executing fund transfers on autopilot. Eric Vaughan, CEO of IgniteTech, said his solution uses grounding—explicit instructions telling the model to escalate for human oversight unless the answer already exists in the company's knowledge base—confining AI to a defined body of knowledge and refusing to answer unless required information is explicitly present, with grounding accuracy running in the mid-90s. Executives can't afford legal gray areas, and the market is responding with systems that keep a specific person accountable for every consequential decision, but only if oversight proves genuine rather than perfunctory. The architecture matters less than whether the human reviewer ever actually overturns the machine.