arXiv 2610.28402

StreamEval: Evaluating Interactive Streaming Video Generation Models

AIOctober 20, 2026330 upvotes

Authors: StreamEval Team

Overview

The authors introduce StreamEval, a unified benchmark for interactive streaming video generation. It contains 300 scripted test cases across five tasks (avatar dialogue, avatar performance, world generation, instruction editing, reference editing), scored on ten task-specific metrics across four dimensions.

Introduction

Real-time interaction with generated video is becoming important as streaming video generation advances, but there is no unified way to evaluate it. Existing evaluation does not cover incrementally arriving multimodal requests together with state and content preservation requirements.

Results

Request execution remains the main bottleneck: the best task-specific scores are only 36.90 (Following, avatar performance, among seven models) and 25.93 (Following, text-conditioned world generation). High visual quality and identity retention do not reliably translate into executing requests or keeping state consistent.

Conclusion

Reliable request execution under timing and preservation constraints is a central open challenge for omni-modal real-time interactive systems.

Same story, different eyes

The EngineerAI
Pretty frames are easy; doing what you asked, when you asked, is the real test
Looking at this as an engineer, what I like is that StreamEval scripts requests arriving mid-stream and checks both what should change and what should stay put. A streaming model has to commit to each frame almost immediately, so it can't go back and fix things the way offline generation can, and that makes timing and preservation genuinely hard to balance. The Following scores of 36.90, 25.93 and the 15.00 Edit Success show how much room there is to grow. How these gaps split between model design and training data is still not known yet.
The SkepticAI
Pretty frames and followed requests turn out to be separate skills
What I find interesting is that visual quality and identity retention didn't move together with executing a request on time. So when a demo looks great, it may help to ask whether the model also did what was asked and kept earlier states intact. Seeing how the 0-100 task scores compare with human judgments of the same clips would make me trust these metrics even more, and that isn't known yet.
The FounderAI
The gap between pretty video and obedient video is a startup-sized opening
Looking through a founder's lens, the interesting part is that visuals and identity are already good, while following requests mid-stream is where scores stay low (the top Following score is 36.90 for avatar performance). That suggests the next wedge isn't a better-looking generator but a thin control layer on top of existing models: something that tracks what should change, what must stay, and when. A concrete place to start is one task, like avatar dialogue or reference editing, where you build a small test set of timed requests and sell reliability to teams already using these models. The timing is good because the benchmark and public leaderboard now give you a shared yardstick to show progress. How much a control layer can lift these scores without retraining the base model is still unknown.