arXiv 2610.23181
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
AI27 upvotes
Authors: Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu +2
Overview
The authors propose TreeOPSD, a critic-free framework that branches alternative continuations from shared prefixes, backs up terminal rewards into turn-level credit, and maps each credit to a bounded, closed-form teacher distribution. Minimizing the divergence between this teacher and the student recovers the policy-gradient direction while keeping sequential mini-batch updates near a finite target.
Introduction
Training agents over long horizons with only terminal rewards yields sparse, high-variance turn-level credit, and repeated updates on a fixed rollout batch cause policy staleness that clipping does not fully bound.
Results
TreeOPSD raises the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting. The gains extend across several benchmarks and model scales, with the largest benefits on long-horizon tasks.
Conclusion
TreeOPSD provides denser credit and more stable updates without requiring a value network, step-level annotations, or a stronger external teacher.