Efficient Training: MoE, Distillation, Compute Trends — 2026-10-14
Recent data reveals a significant shift in AI training economics: post-training and reinforcement learning (RL) compute has overtaken pretraining, while the cost of training frontier models continues to double every eight months. Meanwhile, Chinese tech media reports DeepSeek is raising up to 100 billion RMB to expand its domestic compute infrastructure, signaling a strategic pivot toward self-sufficient, large-scale pretraining despite efficiency gains from Mixture-of-Experts (MoE) architectures.
Efficient Training: MoE, Distillation, Compute Trends — 2026-10-14
Top developments
Post-Training Compute Surpasses Pretraining
As of October 2026, Epoch AI data indicates that compute spent on post-training and RL has officially overtaken pretraining. This shift is reshaping GPU demand and impacting hardware suppliers like Micron, whose High Bandwidth Memory (HBM) sales are driven by the new "reasoning race." This marks a structural change in how labs allocate resources, moving beyond simple scale-up of base models to expensive, iterative reasoning refinement.
DeepSeek Raises Up to 100 Billion RMB for Domestic Compute
DeepSeek is reportedly finalizing a funding round between 80 billion and 100 billion RMB (approx. $11–$14 billion USD), with a pre-money valuation around 500 billion RMB. Unlike previous rounds focused on algorithmic efficiency, this capital is earmarked for "gambling" on domestic compute for pretraining. This move underscores the tension between efficiency techniques like MoE and the sheer brute-force compute required to maintain frontier status under geopolitical constraints.
Return on Token Spend Falls 70% in 2026
New data from Weave's index shows that the return on AI tool spend has dropped significantly. In January 2026, one dollar bought 0.352 expert hours of merged code; by September, it bought only 0.106 hours. This implies the cost per expert hour has risen from $2.84 to $9.39, challenging the narrative that AI efficiency gains are linearly translating to lower operational costs for enterprises.
Inference Costs Hit "One-Hundredth" Era
Reflection AI and other distillation-focused entities are driving inference costs down to one-hundredth of previous levels for specific tasks. This drastic reduction, enabled by advanced distillation models and efficient architectures like Reflection Beam, is changing the competitive landscape for SMEs, making high-quality AI accessible without massive upfront capital expenditure.
Local view
DeepSeek's "Silent" Update and Domestic Compute Push Chinese media outlets including 21st Century Business Herald and Huxiu have focused heavily on DeepSeek’s massive funding round, framing it as a critical bet on domestic compute infrastructure. Simultaneously, community discussions on Weibo and CSDN highlight an unannounced API update by DeepSeek on October 12, where users observed richer output details at slightly slower speeds. This aligns with the broader narrative of DeepSeek prioritizing "engineering cost-performance" to build a sustainable moat amidst sanctions, leveraging its 130 million monthly active users to subsidize compute expansion.
Context & numbers
- Training Cost Growth: Spending on large-scale ML models is growing at 2.4x per year, effectively doubling every eight months. Frontier models now cost hundreds of millions, with projections of $1–3 billion for late 2027 runs.
- Efficiency Gains: Despite cost increases, chip performance per dollar has grown by 0.17 orders of magnitude per year since 2023.
- Model Efficiency: New architectures like Qwen3.8-Next demonstrate that models can achieve near-frontier performance with only 1/9th the training FLOPs and 1/3rd the activated parameters compared to previous generations, highlighting the efficacy of MoE and scaling law optimizations.
On the radar
- DeepSeek V4.1 Flash Speculation: Community tests suggest the recent unannounced API update may be related to a "V4.1 Flash" backend optimization, focusing on output richness over speed.
- Enterprise LLM Buying Trends: G2 reports that 53% of enterprise LLM buyers currently lack formal evaluation frameworks, suggesting a potential market correction as ROI metrics become stricter.
This content was collected, curated, and summarized entirely by AI — including how and what to gather. It may contain inaccuracies. Crew does not guarantee the accuracy of any information presented here. Always verify facts on your own before acting on them. Crew assumes no legal liability for any consequences arising from reliance on this content.
Powered by
