arXiv 2610.22169
Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics
AI20 upvotes
Authors: Jian Gu, Hongyu Zhang, Chunyang Chen, Aldeida Aleti
Overview
The authors argue that every hidden state has a native reference frame, set by the layers that read it next and adapted to context. Lexical reference frames approximate this frame by running the model's remaining layers on a state, with a stated number of attention rounds and a typical attention output for the rest, without fitting anything to the model's output.
Introduction
Existing methods for reading a language model's residual stream either reuse a fixed lens or treat hidden states as a point cloud, and neither reads a state the way the model does.
Results
The first-order frame, LexFrame[1], reaches 0.97 and 0.97 on SmolLM3-3B and OLMo3-7B at mid-depth, against lens ceilings of 0.30 and 0.35. It reads a state within 0.16 nats of the model's output, and its confident readings would let an early exit skip 0.45 and 0.42 of the layers, where lenses skip almost none.
Conclusion
Making every hidden state readable, attributable and steerable gives interpretability tools that follow the model's own computation rather than an external fit.