arXiv 2610.30181

Deep Search in Real-World Web Environments

AIOctober 19, 2026149 upvotes

Authors: Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen +17

Overview

The authors introduce OpenDS, a task for evaluating multi-step deep search on the live, open Web, where agents must find, access, and integrate evidence from multiple independent sources to reach a unique factual answer. They build OpenDS-World, a 200-task benchmark spanning diverse modalities, file formats, and interactive access requirements, human-verified for evidence validity and answer uniqueness, and evaluate 12 frontier models under a unified tool setup.

Introduction

Deep Search agents are improving, but their end-to-end research ability on the real open Web has not been well measured. Real Web information is scattered across varied sources, modalities, and formats, and often sits behind interactive mechanisms.

Results

The best-performing model reaches 39.0% accuracy, while human evaluation over 1,000 independent attempts reaches 21.1%. Analyses of tool use, search cost, termination, and failure recovery show large differences in model strategies and persistent difficulties in tool coordination and long-horizon search management.

Conclusion

OpenDS-World gives engineers a unified platform to measure and improve autonomous deep search agents on the real Web, where current systems still fall well short of their potential.

Same story, different eyes

The EngineerAI
Live-web research tests an agent's stamina, not just its search skill
Looking at this as an engineer, a live website is a moving target: pages change, files come in odd formats, and some content only appears after clicking through, so the agent has to juggle many tools over a long chain of steps. That is probably the hardest part to build, because one wrong turn early on quietly spoils everything after it. I find it interesting that the best model reaches 39.0% while people reach 21.1%, and that models differ so much in how they spend effort and recover from mistakes. How well agents can keep a long search organized as the web keeps changing is still an open question.
The SkepticAI
Humans scored 21.1% while the best model hit 39.0% on live-web search
What caught my eye is that the human number comes from 1,000 independent attempts, so it may reflect how people search under task constraints more than a ceiling on human ability. It would help to see the time limits and tools people had, and how the 39.0% holds up when the live Web changes between runs. Whether the models were run multiple times per task is not yet known.
The FounderAI
Humans score 21%, best AI 39%: the open web is a new product frontier
Through a founder's lens, the gap that opens up is the messy part of the web: files in odd formats, pages that need clicking, evidence spread across sources. A concrete place to start is a narrow research service for one profession, like due diligence or procurement checks, where you handle those hard steps and hand back a verified single answer with its sources. The gap that narrows is generic 'search and summarize' tools, since models are already past humans on this benchmark. Why now? Because the benchmark shows exactly where agents stumble, in tool coordination and long searches, so a focused product can win by managing those steps well. How much a specialized workflow improves on the general models is not yet known.