Grafana Labs is adapting a familiar reliability engineering technique to solve one of the trickier questions in modern software: whether an AI agent is actually doing its job well. In a recent article, the company lays out an approach that uses evaluations, service-level objectives, and error budgets to transform behaviors like hallucination, task completion, toxicity, and cost into trackable targets that teams can measure over time. The concept tackles a core weakness in traditional application monitoring—conventional metrics can confirm that an agent replied fast, hit no technical snags, and burned through a certain token count, but they can't say whether the answer was right, grounded in real information, or helpful to the person asking.
Grafana's method starts with evaluation, where each agent interaction gets checked against specific behavioral standards. The company describes tests such as verifying whether a response stayed grounded in its source material, satisfied the user's request, included toxic language, leaked personal details, or fell victim to prompt injection. These evaluations can rely on another language model acting as a judge, though deterministic tools like regular expressions work when the behavior can be spotted without an LLM. Once a behavior becomes measurable, standard SRE frameworks kick in—Grafana gives the example of requiring 95% of conversations to be judged as fulfilled across a 30-day window, with the leftover 5% forming the error budget: the volume of unacceptable behavior the team has consciously agreed it can live with during that stretch. The company's existing SLO tooling follows the same pattern, defining an SLO through a service-level indicator, target, and time frame, with the error budget calculated from the gap between the target and 100%.
The report emphasizes that quality can't be boiled down to a single number. An agent might be well grounded but expensive, helpful but occasionally toxic, or accurate but slow, and treating these as separate measurements lets teams set different thresholds for the behaviors that matter to a given application rather than burying important failures inside an averaged score. According to Grafana, the approach becomes especially valuable when agent behavior degrades slowly instead of breaking outright—a model swap, prompt tweak, retrieval adjustment, or shift in how users interact might cause quality to slip by a fraction of a percentage point each day, and monitoring error-budget burn makes that kind of drift visible. The report also notes a key limitation: an SLO is only as useful as the measurement behind it, meaning organizations still need representative evaluation datasets, clear definitions of acceptable behavior, appropriate deterministic checks where feasible, human review for edge cases, and ongoing validation of the evaluators themselves.
The significance lies in extending a mature SRE principle from system reliability to behavioral reliability. Grafana argues that an agent can be technically up while becoming operationally unreliable because its answers grow steadily more inaccurate, ungrounded, expensive, or unsafe, and treating those behaviors as measurable service objectives gives engineering teams a way to spot that degradation and decide when experimentation should pause in favor of fixing quality. The company's work fits into a wider industry push—Anthropic has built formal evaluation practices using defined tasks, repeated trials, and multiple graders, while OpenTelemetry is working on standardized semantic conventions for agent telemetry so traces, metrics, and logs provide a consistent foundation for evaluating agent behavior across frameworks and vendors. Together, these efforts sketch an emerging pattern: evaluations determine whether an agent behaved correctly, observability supplies the evidence, and SLOs set the operational threshold for deciding whether that behavior is acceptable. The real shift isn't just about adding another AI metric—it's recognizing that systems can stay online while quietly becoming less trustworthy, and giving teams the governance tools to catch that slide before it becomes a crisis. For organizations putting AI agents into production, the ability to separate technical uptime from behavioral reliability may prove to be one of the more critical distinctions in how they manage risk and trust at scale.