Archive — CN — technode
WeChat AI Team Details WeLM Models Scaling to 617B Parameters
Tencent's WeChat AI team unveiled their scaling strategy for WeLM models, introducing WeLM-HD4-80B and WeLM-HD4-617B using Hidden Decoding, enabling significant parameter activation and performance improvements over baselines.
Key points
- Tencent announced new WeLM models: WeLM-HD4-80B with 80 billion parameters and WeLM-HD4-617B with 617 billion parameters.
- The method used is called Hidden Decoding, which expands tokens into multiple computation streams without altering the Transformer backbone.
- The 80B model activates 3 billion parameters, while the 617B model activates 23 billion parameters.
- Both models surpassed autoregressive baselines in nine benchmarks.
- Training costs were notably high: 5.1 times baseline for the 80B model and 4.4 times for the 617B model.
Why it matters
- This advancement reflects the growing trend towards increasingly larger AI models, paralleling developments in the field like OpenAI's GPT advancements.
- By 2025, there will be a stronger focus on efficiency and cost-effectiveness in AI model training, as seen in the high training costs reported.
Tencent's innovative scaling of WeLM models exemplifies the current trend in AI toward massive parameter models and highlights the significant resources required for leading-edge AI development.