arXiv 2610.23479

HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents

RoboticsOctober 16, 202684 upvotes

Authors: Hongcheng Gao, Alex Zhou, Jay Zhu, Zelin Zheng, Shijia Ge, Xiping Li, Di Wu, Yazhe Wang +8

Overview

The authors test how far a vision-language-model coding agent can go when it controls a two-arm robot through a command line instead of a computer. The agent reads three calibrated RGB cameras, and after each failed attempt it writes or fixes small tools that use triangulation and plane intersection to locate objects. The tools are built on ten layouts per task, frozen, and then tested on ten new layouts.

Introduction

VLM agents now compete with learned policies on robot manipulation leaderboards, but it is unclear how far a coding agent can go when it controls a robot. The paper asks where such an agent stops working.

Results

HexaShell reaches a 39.7% success rate and a progress score of 48.0 across all 42 RoboDojo tasks, a new state of the art on that benchmark. Pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%, which suggests contact, not reasoning, is the main limit.

Conclusion

The results show that a thin harness with evolving code tools can carry a coding agent into robot control, and that contact-rich manipulation remains the key bottleneck.

Same story, different eyes

The EngineerAI
A robot that saves its lessons as code, like fixing a failing test
What I find charming is that the agent treats a missed grasp like a failing test: it reads the log, finds the cause, and patches a small tool. Triangulation from the cameras (or a known table plane) gives it object positions without any learned model, and a wrist-camera second view fills in the gaps. The hard part for the builders was probably contact, since the agent can often see why an insertion failed but still can't make the motion work. How tools could carry that fine force-and-feel knowledge is not yet known.
The SkepticAI
Tools tuned on 10 layouts, then tested on 10 fresh ones
What I find nice here is the setup: the tools get frozen after being built on ten layouts, then are tried on ten new ones of the same seed. So the 39.7% success rate reads as how well the saved code carries over. It would be even more reassuring to see how it holds up on a different seed or a different scene, which hasn't been reported yet. Since the tools are open-sourced, readers could also rerun them themselves to see how the 8.3% on tight insertion compares with the 92.5% on pushing.
The FounderAI
Robot skills are becoming code you can version, share, and reuse
What caught my eye is that the learning lives in small tools written as code, not in model weights, so it can be reused and improved by anyone. That opens room for a marketplace or a maintenance business around tool libraries for specific tasks like pushing, sorting, and memory-based handling. The gap that stays open is contact-heavy work like tight insertion, where success is only 8.3%, so a startup could begin with a narrow task and pair it with a force-sensing wrist or a better gripper. It feels like the right moment because a thin harness plus an off-the-shelf coding agent already reaches 39.7% without any robot-specific training, and the tools are open source to start from.