JioHotstar tested six different methods for helping AI coding agents work through large codebases and discovered that no single approach consistently outperformed the others across all programming tasks. In a detailed evaluation published on the company's engineering blog, Senior Staff Engineer Abhishek Bansal found that structural code search excelled when engineers faced well-defined problems, while semantic retrieval proved more valuable when requirements were vague or open-ended. The study also revealed that layering retrieval systems on top of basic tools didn't reliably boost task completion rates or cut down on token usage.
The team compared six approaches: filesystem tools, Sourcebot, Graphify, CocoIndex Code, Repowise, and an internally built retrieval-augmented generation pipeline. They measured performance using portions of SWE-bench and CodeScaleBench, plus a custom question-answering test built around JioHotstar's own codebase. All experiments ran on Claude Sonnet and tracked task accuracy, token consumption, agent turns, and how easy each tool was to use. The filesystem baseline, which relied on standard search and file operations in locally checked-out repositories, stayed competitive on concrete tasks. Sourcebot, offering indexed text search, symbol definitions, references, and file reads, beat the baseline on tougher SWE-bench Verified tasks while also costing less. On easier SWE-bench Lite tasks, retrieval methods generally created extra overhead without improving results. CodeScaleBench showed a different trend: semantic retrieval helped agents interpret unclear problem statements and connect them to implementation details that weren't obvious from exact text or symbol searches.
According to the report, retrieval approaches should be chosen based on task characteristics rather than treated as one-size-fits-all solutions. The study also stressed the importance of tool design, noting that agents frequently favored familiar developer operations like searching text, reading files, and locating symbols over higher-level abstractions that required additional verification. Siddharth Vaithinathan raised a broader architectural question in a LinkedIn post discussing the study, arguing that repository indexing alone can't capture why architectural decisions were made, which constraints shaped them, or what earlier incidents taught the engineering team. He suggested extending agent knowledge beyond code and documentation to include architecture, constraints, decisions, rationale, incidents, and outcomes.
The report explains that structural search works well for concrete, code-focused problems because it can pinpoint exact text matches and symbol definitions efficiently. Semantic indexing, by contrast, helps when agents must bridge the gap between ambiguous requirements and implementation specifics that don't appear in literal searches. This split reflects a fundamental trade-off: structural approaches deliver precision when the target is clear, while semantic methods offer flexibility when the problem statement is fuzzy. The difference matters because adding retrieval layers increases complexity and token overhead, which only pays off if the task actually needs the extra interpretive power.
JioHotstar recommends keeping filesystem tools as a baseline, deploying structural search for concrete, code-centric problems, and turning to semantic indexing for ambiguous or cross-cutting requirements. The engineering team also stresses measuring tool-use traces alongside task accuracy to see whether retrieval cuts down on unnecessary exploration. The takeaway is clear: match retrieval capabilities to the programming task and how the agent navigates the codebase, rather than assuming more sophisticated tools always deliver better results. The tension between precision and flexibility in code navigation tools mirrors a wider challenge facing engineering leaders: deciding when automation should replicate expert judgment versus when it should simply accelerate routine work.