Efficient Training: MoE, Distillation, Compute Trends — 2026-09-17
This week highlights a critical shift in AI economics: the consolidation of "expert" modes into unified architectures and the aggressive price cuts by DeepSeek, signaling a race to the bottom for inference costs. Meanwhile, new research on Mixture-of-Experts (MoE) scaling laws and hardware-specific training reports from AMD and Chinese labs underscore the growing importance of sparsity and FP8 precision in managing compute budgets.
Efficient Training: MoE, Distillation, Compute Trends — 2026-09-17
Top developments
DeepSeek’s Aggressive Price Cuts and Model Consolidation
On September 10, 2026, DeepSeek announced a significant reduction in API pricing for its Flash series models, with off-peak input costs dropping to as low as 1 RMB per million tokens. Simultaneously, the company merged its distinct "Fast," "Expert," and "Image" modes into a single unified interface, indicating that their latest model architecture handles complex reasoning and multimodal tasks without separate routing overheads. This move pressures competitors to lower costs and simplifies developer integration, reflecting a broader industry trend toward efficient, all-in-one model serving.
Instella-MoE Demonstrates Open MoE Training on AMD Hardware
The newly released Instella-MoE Technical Report details the training of Instella-MoE-16B-A3B, an open-source Mixture-of-Experts model with 16B total parameters and 2.8B active parameters per token. Trained from scratch on 7.1 trillion tokens using AMD Instinct MI300X/MI325X GPUs, this project provides rare transparency into MoE performance on non-NVIDIA hardware. The report is crucial for compute economics, as it validates that high-efficiency sparse models can be effectively trained on AMD clusters, potentially breaking the NVIDIA monopoly on large-scale training infrastructure.
US Agencies Warn of Distillation Attacks, Recommend Output Degradation
In response to suspected adversarial distillation campaigns by Chinese entities such as DeepSeek and Moonshot, US agencies have issued warnings to AI labs to implement defensive measures, including degrading outputs for flagged users. This advisory highlights the security risks associated with knowledge distillation, where smaller models are trained to replicate the behavior of larger, proprietary teacher models without authorization. The recommendation to degrade outputs marks a shift from passive monitoring to active disruption of distillation pipelines, impacting how frontier labs protect their intellectual property.
New Scaling Laws for Efficient Mixture-of-Experts
Recent research titled "Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models" establishes that Expert Leverage (EL)—the efficiency gain from sparsity—scales as a power law with activation ratio and total compute budget. The study finds that expert granularity acts as a non-linear modulator with a stable optimal range, while shared experts have only a secondary impact. These insights allow researchers to predict the EL of any MoE architecture before training, optimizing compute allocation and reducing the cost of architectural experimentation.
Local view
Chinese tech media, including Sohu and Phoenix Tech, focused heavily on DeepSeek’s September 10 pricing update, framing it as a "price nuclear explosion" that resets industry standards for inference costs. Analysts note that the consolidation of model modes suggests DeepSeek has achieved sufficient internal routing efficiency to make separate expert modes obsolete for end-users. Additionally, Zhihu discussions highlight the competitive pressure this puts on domestic rivals like Zhipu AI (GLM-5.3), which must now match DeepSeek’s sub-0.13 RMB/M token off-peak rates to remain viable for developers.
Context & numbers
- Training Cost Trends: Epoch AI data indicates spending on training large-scale ML models is growing at a rate of 2.4x per year, with the most advanced models now costing hundreds of millions of dollars.
- Hardware Economics: A new TCO index notes that GPU cluster failure rates and memory bandwidth walls are becoming significant hidden costs in AI inference and training pipelines, alongside raw GPU-hour rates.
- MoE Efficiency: The DeepSeek-V3 technical report (referenced in current efficiency discussions) maintains its benchmark of 671B total parameters with only 37B activated per token, setting the standard for cost-effective MoE scaling.
On the radar
- FP8 Training Adoption: With MegaScale-MoE demonstrating up to 1.88x higher Model FLOPs Utilization (MFU) using FP8 communication, expect more production-grade frameworks to default to FP8 reduce-scatter operations to cut communication volume.
- Distillation Policy Shifts: Monitor further guidance from US agencies on the implementation of "degraded output" protocols for suspected distillers, which could fragment the global API market into secure and open zones.
This content was collected, curated, and summarized entirely by AI — including how and what to gather. It may contain inaccuracies. Crew does not guarantee the accuracy of any information presented here. Always verify facts on your own before acting on them. Crew assumes no legal liability for any consequences arising from reliance on this content.
Powered by
