Inference Efficiency: MLPerf, Tokens per Dollar, Hardware — 2026-10-23
DeepSeek’s open-sourcing of its DeepInfer framework has triggered a re-evaluation of inference economics, revealing $38/hour revenue per GPU and potential margins exceeding 95%. Meanwhile, Inception Labs’ Mercury 3 model sets a new speed baseline at 1,068 tokens per second, challenging traditional autoregressive architectures. These developments coincide with new data on GPU revenue ladders and the ongoing externalization of TPU inference stacks.
Inference Efficiency: MLPerf, Tokens per Dollar, Hardware — 2026-10-23
Top developments
DeepSeek Discloses $38/Hour GPU Revenue and 95% Margins
On October 20, DeepSeek fully open-sourced its internal inference framework, DeepInfer, and the token codec component deepseek-recipe. The disclosure revealed that a single GPU card generates $38 in hourly revenue, with some analysts estimating inference gross margins between 84.2% and 96.1%. This framework is compatible with both NVIDIA GPUs and Huawei Ascend NPUs, supporting a single-cluster daily throughput of 155 trillion tokens. These figures suggest that optimized software stacks can significantly lower the effective cost per token, potentially sparking price wars among API providers.
Inception Mercury 3 Hits 1,068 Tokens/Second with Diffusion Architecture
Released on October 19, 2026, Inception Labs’ Mercury 3 model achieved 1,068 tokens per second on NVIDIA GPUs, establishing a new performance baseline for the "fast-cheap" tier of frontier models. The model utilizes a diffusion-based architecture to generate tokens in parallel, bypassing the sequential bottlenecks of traditional autoregressive decoding. This speed renders previous high-speed models obsolete and highlights the potential of non-autoregressive approaches for latency-sensitive applications.
GPU Revenue Ladder: From $38 Theoretical to $2.44 Realized
Recent analysis by Tomasz Tunguz contrasts DeepSeek’s theoretical $38/hour GPU revenue with SpaceX’s realized $2.44/hour, illustrating the vast disparity in inference monetization efficiency. The data suggests that while payback periods on GPU investments can be under a year, the 2027 supply pipeline remains largely unbuilt, creating a complex landscape for hardware ROI. This "revenue ladder" pricing four different scenarios highlights the gap between peak theoretical utilization and actual market conditions.
Local view
Chinese media outlets, including 21st Century Business Herald and Sina Finance, have extensively covered DeepSeek’s open-sourcing of DeepInfer, framing it as a strategic move to lower AI service costs and energy consumption. Reports highlight that DeepSeek’s inference efficiency exceeds most competitors, with data from SemiAnalysis showing high throughput on vLLM frameworks. Local stakeholders are analyzing the compatibility of DeepInfer with Huawei Ascend chips as a key factor in domestic AI infrastructure resilience.
Context & numbers
- Throughput: DeepSeek’s single cluster processes 155 trillion tokens daily.
- Speed: Inception Mercury 3 achieves 1,068 tokens per second.
- Revenue: DeepSeek GPU hourly revenue is $38; SpaceX realized revenue is $2.44.
- Margins: Estimated inference gross margins for DeepSeek range from 84.2% to 96.1%.
On the radar
- TPU Externalization: SemiAnalysis notes that Google’s TPU InferenceX stack is seeing rapid externalization, with claims of up to 50% better performance per dollar compared to traditional CUDA-based stacks.
- ASIC Bargaining Power: Analysts suggest that the mere presence of credible custom chips (ASICs) is enough to lower NVIDIA’s pricing power, even if parity is the only goal.
- Groq Pricing: Groq continues to compete on price with rates starting at $0.05 per million input tokens, challenging standard GPU-based API providers.
This content was collected, curated, and summarized entirely by AI — including how and what to gather. It may contain inaccuracies. Crew does not guarantee the accuracy of any information presented here. Always verify facts on your own before acting on them. Crew assumes no legal liability for any consequences arising from reliance on this content.
Powered by
