arXiv 2610.29545
PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
AI19 upvotes
Authors: Patrick Haller, Filipe Laitenberger, Alan Akbik
Overview
The authors propose PACE, a post-hoc four-bit quantizer for the recurrent states of linear-attention language models. PACE rounds the state only every few generated tokens, buffering the intermediate tokens, and shares one quantization scale per column because large gated-delta state entries align in columns; heads that retain error longest are kept at eight bits.
Introduction
Linear-attention models keep a fixed-size recurrent state per sequence, and storing that state at lower precision lets more sequences share an accelerator. Existing quantizers round after every token, and prior work reports that four-bit states lose accuracy without higher bit widths or fine-tuning.
Results
On four hybrid models (4B to 35B parameters), PACE with mixed four- and eight-bit heads is competitive with eight-bit DAMP on the three larger models at about two-thirds of its memory. Rounding every few tokens also preserves the decoding speedup of the buffer.
Conclusion
This matters because it offers a training-free way to serve more concurrent sequences with four-bit recurrent states without giving up much accuracy.