arXiv 2610.09872

LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

AIOctober 7, 202640 upvotes

Authors: Jun Zhao, Leiming Fu, Yanbo Wen, Yiding Wang, Xuantong Liu, Yang Shu, Yuyang Lu, Xuanran Xing +4

Overview

The authors built LiveMACEBench, a benchmark that uses live financial markets to test persistent LLM agents. Five frontier LLMs ran continuous trajectories over 30 days under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. Each agent was assessed on realized outcomes and on mechanism-specific diagnostics drawn from complete decision traces.

Introduction

Judging agents only by outcomes can hide the capabilities that produce them, a problem especially pronounced in evolving environments where results reflect a closed-loop interaction with changing external conditions.

Results

The most important finding is a pronounced outcome-capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from very different patterns of mechanism use. Trace-level diagnostics also reveal distinct bottlenecks across capabilities, showing that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures.

Conclusion

LiveMACEBench turns live markets from a performance leaderboard into a diagnostic environment that makes this distinction measurable for agent capability.

Same story, different eyes

The EngineerAI
Live markets can show how an agent works, not just whether it wins
What I like here is that the agent's whole decision trace gets recorded, so you can ask whether it actually used its memory, tools, or rules, and not only whether it made money. In a live market, luck and timing can hide a lot, so separating 'had access to a mechanism' from 'used it well' seems like the hardest part to get right. It's also still unclear how the findings from these 30 days will carry over to longer periods.
The SkepticAI
Similar returns can hide very different ways agents work
What I find interesting is that the benchmark separates three things: having a tool or memory, actually using it well, and the final return. If you want to trust a headline result, it helps to check whether agents with similar returns used those mechanisms in similar ways. It would also be reassuring to see whether the per-capability diagnostics hold up beyond 30 days and across different market conditions. How much of the return gap comes from skill versus market luck is not yet known.
The FounderAI
Agent evaluation is its own product: diagnosing why, not just who won
If returns and capability scores can diverge, then a leaderboard alone leaves a gap, and that gap looks like a real opening. A good first step is a small diagnostic layer that records an agent's decision traces and shows which mechanism (memory, tools, rules, collaboration) is the bottleneck, starting in one narrow domain like trading or ops. The timing feels right because teams are putting persistent agents into live settings now and need to know what to fix, not just who ranked first. Meanwhile, the window for products that only show outcome rankings may be narrowing. How much teams will pay for this kind of diagnosis is not yet known.