arXiv 2610.21082
SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
AI11 upvotes
Authors: Youran Sun, Xingyu Ren, Chugang Yi, Kejia Zhang, Di Wu, Yingkang Cao, Allen Zhou, Jianping Ye +10
Overview
The authors built SciΠvot, a 248-case benchmark from six months of records in which senior researchers ran LLM agents on 26 projects across six scientific fields and corrected their research judgments. In each case, a model chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction. They then compared model and human performance under different reasoning efforts, with and without the project record, and with reconsideration.
Introduction
LLM agents can run scientific experiments for months with little supervision, but reliable autonomous research also requires sound scientific judgment, which is hard to evaluate because research decisions are open-ended and context-dependent.
Results
At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative, and four of five frontier models perform below chance on the selected cases. The best project-condition accuracy is 63.5%, compared with 73.4% for fresh researchers.
Conclusion
SciΠvot provides a systematic way to measure whether LLM agents can make the scientific judgments that autonomous research requires.