Netflix has open-sourced an agentic workflow designed to reduce manual effort in observational causal inference, a statistical method used to estimate cause-and-effect relationships from data that wasn't collected through controlled experiments. In a case study examining how new entertainment types like games affect user retention, the agent-driven workflow produced an estimated effect just 25% as large as a baseline result from a standard language model using simple linear regression. The system aims to automate tedious or error-prone steps like sensitivity checks and iteration tracking, while leaving high-level work—such as question formulation and result evaluation—to human analysts.
The workflow relies on two separate agents working in tandem: an actor that generates specifications, populates parameters in templated Jupyter notebooks, and runs the analysis, and a critic that reviews the output and assigns one of three ratings (not_satisfactory, satisfactory_with_caveats, or fully_satisfactory) before recommending changes to the specification. Netflix tested the system on the Atlantic Causal Inference Conference competition dataset and reported that performance was "competitive" against benchmark approaches. In the retention case study, the critic flagged multiple problems including possible early adopter bias and a placebo test failure, issues the workflow's built-in playbooks then addressed through automated reruns with adjusted parameters. The analysis plan defined "treatment" as days spent engaging with the new content type and "outcome" as two-month retention.
According to Netflix, "Using agents for causal inference poses a challenge: how do we evaluate agents' performance on tasks without ground truth?" To address this, the workflow combines process audits—where agents publish plans, specifications, plots, and notebooks that humans can inspect and rerun—with direct human oversight. The company states that it open-sourced a lightweight, standalone version of the system "to enable others to learn from and critique our workflow," hoping the release will "stimulate more research and development on agentic evaluation in the absence of ground truth." Fabio Piazza, a senior product manager at Owkin, noted on LinkedIn that Netflix "continues to quietly set the bar on applied AI" by making every step transparent rather than only checking final outputs.
The framework treats observational causal inference as target trial emulation, essentially searching for the best hypothetical A/B test design to answer a given question. Because many causal analyses lack a definitive right answer, Netflix's approach emphasizes auditable intermediate steps instead of relying solely on end results. Taikai Takeda, a director of software engineering at Indeed.com, observed that without such structure, a casual user might "just casually throw it at an LLM" and end up satisfied with "some half-baked regression analysis," whereas this system is "designed not so much for the agent to spit out answers, but to guide through the correct steps while leaving artifacts that experts can verify later." The workflow builds on existing observational causal inference tools already in use at Netflix, extending them with agent capabilities.
Netflix published the oci-agent source code on GitHub, providing experiments on public datasets and detailed case studies to help others adopt or adapt the methodology. The company frames the release as part of ongoing work to balance automation with human judgment in statistical analysis, especially for tasks where definitive ground truth isn't available. By splitting responsibilities across actor and critic agents and documenting every decision, the system aims to make causal inference more accessible without sacrificing rigor. For teams that lack deep causal inference expertise, the dual-agent structure and built-in playbooks offer a guardrail against common pitfalls—turning what might have been a misleading simple regression into a multi-step, auditable analysis that surfaces its own limitations. The real test will be whether open-source adoption leads to workflow improvements that extend beyond Netflix's internal use cases, or whether the need for domain-specific playbooks limits broader applicability.