arXiv 2610.27598

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

AIOctober 20, 202625 upvotes

Authors: Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang, Haozheng Hou, Yicheng Gu, Hanjing Wang, Qirui Zeng +6

Overview

The authors compare Local Experts (layer-specific, as in mainstream MoE models) against proposed Meta Experts that are shared across layers. They run a multi-dimensional empirical study and add a theoretical analysis of residual pathways.

Introduction

Mainstream MoE models stack layer-specific experts that often store highly similar, redundant content, wasting parameters. The question is whether sharing experts across layers can reduce this redundancy and scale MoE more efficiently.

Results

With activated experts and granularity fixed, the Meta Expert model matches Vanilla MoE while using only 70–80% of the experts. It also shows lower parameter redundancy, lower cross-entropy loss, and higher downstream accuracy than Local Expert models.

Conclusion

Cross-layer expert sharing offers a practical way to cut expert storage and improve MoE scaling without sacrificing performance.

Same story, different eyes

The EngineerAI
Experts shared across layers could make MoE storage leaner
From an engineer's view, the appealing part is that one expert can be reused by several layers, so the model stops storing near-duplicate knowledge in every layer. It also gives hidden states more routes to reach what they need, which feels like a nice bonus beyond just saving memory. The hard part is probably routing, since layers now compete for the same experts and training has to keep them balanced. How well this holds up at much larger scales is not yet known.
The SkepticAI
Sharing experts across layers: the same quality with 70–80% of the experts
What I find interesting is that the 70–80% figure comes from holding the number of activated experts and expert granularity fixed, so the comparison looks like-for-like. To trust it a bit more, it would help to see whether the savings hold at larger model sizes and across different training data, and whether total parameters and inference speed also drop in practice. The retrieval argument, that hidden states get more flexible residual pathways, rests on theory, so a small ablation that tests that mechanism directly would make it more convincing. How the routing cost changes when experts are shared across layers is not yet known.
The HistorianAI
Sharing experts across layers echoes the shared-library move, with a twist on what gets shared
Giving every layer its own experts feels like the era when each program carried its own copy of a library, and shared libraries saved memory by removing that duplication. Here, sharing experts across layers matches Vanilla MoE with only 70–80% of the experts. What strikes me is the contrast with ALBERT, which shared whole layers and so tied every layer to the same function, whereas here the router can pick a different combination of experts at each layer. That flexibility may be what keeps the cost of sharing low, though whether it holds at much larger scales isn't known yet.