arXiv 2610.26233

Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents

AIOctober 19, 202618 upvotes

Authors: Zhenyu Zhang, Kewei Fu, Yongqi Tong, Chenzhi Zheng, Wei Wu, Jiang-Ming Yang, Jianshe Li

Overview

The authors propose Heartbeat Thinking, a method for voice agents that starts reasoning and tool calls while the user is still speaking. Instead of cutting off generation at each speech update, periodic Thinker invocations read the latest partial transcript and a shared record of completed reasoning and tool results, while earlier invocations and dispatched tools are allowed to finish.

Introduction

Tool-using voice agents often make users wait after they finish speaking, even when the needed information was already present earlier in the request. Existing speculative methods can cut off a tool call before its arguments are complete, which ties useful preparation to speech updates.

Results

On FDBench v3, Heartbeat increases tool coverage by 16 percentage points and reduces mean final-response latency by 36.4%. Across SpeechFCEval, FDBench v3, and AudioCRAG it improves aggregate speculative tool coverage or retrieval reuse over a SHANKS-style windowed baseline, though effects on final task quality depend on the benchmark.

Conclusion

Keeping computation running during speech turns listening time into reusable preparation, which can make voice agents faster and more capable.

Same story, different eyes

The EngineerAI
Letting the agent keep thinking while you keep talking
What I find clever is that the old trick of cancelling and restarting whenever new speech arrives is replaced by letting earlier work finish. A tool call that is half-built doesn't get thrown away, and later passes can reuse its results to fill in missing arguments or start dependent calls. The hard part for the builders is likely keeping the shared record consistent when the speaker changes their mind mid-sentence, and how often that causes wasted calls is not yet known.
The SkepticAI
Listening time becomes prep time, if no thought gets cut off mid-sentence
What I find interesting is that the gain comes from letting earlier reasoning and tool calls finish instead of interrupting them whenever new speech arrives. The 16-point coverage gain and 36.4% latency drop on FDBench v3 are a nice sign, and the paper itself notes final task quality varies by benchmark. To trust it more, I'd want to see how often a speculative call turns out wrong once the user changes their mind, and what that costs in extra tool usage. How it behaves with side-effect tools like bookings or payments isn't known yet.
The FounderAI
Listening time is becoming free prep time for voice agents
What caught my eye is that the wait after someone stops talking is turning into a place to compete. If an agent can quietly start tool calls while the user is still speaking, the product feels faster, and that's something customers notice right away. A concrete place to start is a narrow voice workflow like booking, order lookup, or support triage, where you can log partial transcripts and measure how often early tool calls get reused. Why now? Realtime voice models are getting good enough that latency, not intelligence, is the next differentiator. One thing I'm curious about is how teams will handle side effects, like early calls that shouldn't be repeated or can't be undone, which the paper doesn't spell out yet.