DeepSeek-V4: Breaking the Efficiency Barrier in Million-Token AI
Imagine trying to read a book with a million pages, but you can only see one word at a time. That is the fundamental challenge that has plagued large language models when dealing with ultra-long contexts. The attention mechanism, which is the heart of modern AI, grows quadratically in complexity as the input length increases. A 1,000-token prompt is manageable. A 10,000-token prompt starts to hurt. A 1,000,000-token prompt is a computational nightmare.
For years, the AI community has grappled with this problem. The emergence of reasoning models has established a new paradigm of test-time scaling, where models spend more computational effort during inference to achieve better results. But this scaling has been fundamentally constrained by the quadratic cost of attention. Every extra token of reasoning costs exponentially more. This has been a major bottleneck for long-horizon tasks like agentic workflows, massive cross-document analysis, and complex multi-step problem solving.
Enter DeepSeek-V4. In March 2026, DeepSeek AI released a preview version of their next-generation language model series, and it fundamentally changes the game. The DeepSeek-V4 series introduces a hybrid attention architecture that dramatically reduces the computational cost of processing ultra-long sequences. The result is a model that can efficiently handle a million tokens of context while consuming only a fraction of the resources required by previous models.
In this comprehensive guide, we will explore the DeepSeek-V4 series from every angle. We will look at the problems it solves, the architectural innovations that make it possible, the training infrastructure that supports it, the remarkable performance it achieves on real-world tasks, and the DSpark speculative decoding framework that makes deployment practical. Whether you are an AI researcher, a machine learning engineer, or simply someone interested in the cutting edge of technology, this guide will take you through everything you need to know about DeepSeek-V4.
Let us begin.
Chapter 1: The Problem – Why Long Contexts Are Hard
The Quadratic Bottleneck
The Transformer architecture, introduced in 2017, has been the foundation of nearly every major language model since. Its core innovation is the attention mechanism, which allows each token in a sequence to attend to every other token. This is powerful because it enables the model to capture long-range dependencies and understand context in ways that previous architectures could not.
However, there is a catch. The computational cost of attention grows quadratically with the sequence length. If you double the length of your input, you quadruple the amount of computation required. For short sequences, this is manageable. For sequences of 1,000 or 2,000 tokens, it is fine. But as models have grown more powerful and tasks have become more complex, the demand for longer contexts has increased dramatically.
Consider a model with a 1 million token context window. In a standard attention mechanism, each of the 1 million tokens would need to attend to every other token. That is 1 trillion attention pairs. The memory required to store the attention matrices alone would be measured in terabytes. The computational cost would be astronomical.
This is not just a theoretical problem. It has real-world consequences. Researchers have been exploring test-time scaling, where models spend more computational effort during inference to reason through complex problems. But this scaling is fundamentally limited by the cost of attention. If every additional reasoning token costs exponentially more, there is a hard ceiling on how much test-time scaling can achieve.
The Rise of Long-Horizon Tasks
At the same time, the AI community has been moving toward longer and more complex tasks. Agentic workflows, where AI agents perform multi-step tasks across extended time horizons, require models to maintain context over thousands or even millions of tokens. Cross-document analysis, where a model must synthesize information from dozens of documents, demands the same.
The DeepSeek-V3 series, released in late 2024, was already a significant step forward. It introduced Multi-head Latent Attention (MLA) and other optimizations that improved efficiency. But even DeepSeek-V3.2, with its 671 billion total parameters and 37 billion activated parameters, struggled with ultra-long contexts. In the 1-million-token setting, the computational cost was simply too high for practical deployment.
DeepSeek-V4 was designed to break this barrier. The goal was ambitious: achieve native, efficient support for million-token contexts while maintaining or improving model capability. And as the results show, they succeeded.
Chapter 2: The Architecture – A Hybrid Approach to Attention
The Core Innovation: CSA and HCA
The most significant architectural innovation in DeepSeek-V4 is the hybrid attention mechanism, which combines two complementary approaches: Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). These are not used together in every layer. Instead, they are interleaved across the model's layers, creating a hybrid architecture that balances efficiency and expressivity.
Compressed Sparse Attention (CSA)
CSA is the more sophisticated of the two approaches. It operates in two stages: compression and sparse selection.
Stage 1: Compression. CSA first compresses the Key-Value (KV) cache of every m tokens into a single entry. In DeepSeek-V4, m is set to 4. This means that every 4 tokens are condensed into one compressed KV entry. The compression is not a simple average. It uses learned compression weights and positional biases to intelligently combine the information from the 4 tokens into a single, information-rich representation.
The compression process is overlapped, meaning that the tokens used for one compressed entry overlap with the tokens used for the next. This creates a smooth, continuous compression that preserves more information than non-overlapping compression would.
Stage 2: Sparse Selection. After compression, CSA applies DeepSeek Sparse Attention (DSA). Instead of having each query token attend to all compressed KV entries, each query only attends to the top-k most relevant compressed entries. The selection is performed by a Lightning Indexer, which is a lightweight mechanism that computes relevance scores between queries and compressed KV entries.
The indexer works by producing indexer queries from each token and comparing them to compressed indexer keys. The top-k compressed entries with the highest scores are selected for core attention. This sparse selection dramatically reduces the number of attention pairs that need to be computed.
Sliding Window Attention. CSA also includes an additional branch of sliding window attention. For each query token, the model attends to the most recent n_win tokens (128 in DeepSeek-V4) in addition to the selected compressed entries. This ensures that local dependencies are preserved, which is important for language modeling where recent tokens often have the greatest relevance.
Heavily Compressed Attention (HCA)
While CSA uses moderate compression (m=4) and sparse selection, HCA takes a different approach. It applies much more aggressive compression (m'=128) but keeps the attention dense. Every query token attends to all compressed KV entries.
The logic is straightforward: if you compress the sequence length by a factor of 128, the cost of dense attention on the compressed sequence is only 1/128th of the cost of dense attention on the original sequence. This makes dense attention on the compressed sequence computationally feasible.
HCA does not use an indexer or sparse selection. It simply compresses the KV cache and performs dense attention on the compressed representation. This is simpler and faster than CSA, but it also discards more information. The tradeoff is handled by interleaving CSA and HCA across layers.
How They Work Together
CSA and HCA are interleaved across the layers of the model. In DeepSeek-V4-Pro, which has 61 layers, the first two layers use HCA. The remaining layers alternate between CSA and HCA. This interleaving ensures that the model benefits from both approaches: CSA provides fine-grained, context-aware attention with moderate compression and sparse selection, while HCA provides a global, heavily compressed view of the entire context.
The hybrid approach delivers dramatic efficiency gains. In the 1-million-token context setting, DeepSeek-V4-Pro requires only 27% of the single-token inference FLOPs and 10% of the KV cache compared with DeepSeek-V3.2. For DeepSeek-V4-Flash, the numbers are even more impressive: 10% of the FLOPs and 7% of the KV cache.
Manifold-Constrained Hyper-Connections (mHC)
In addition to the hybrid attention mechanism, DeepSeek-V4 introduces Manifold-Constrained Hyper-Connections (mHC). This is an upgrade to conventional residual connections, which are a fundamental building block of Transformer models.
Residual connections allow gradients to flow directly through the network without passing through every layer. This helps with training stability and enables the training of very deep networks. However, standard residual connections have limitations. They can cause signal propagation issues in very deep networks, and they do not provide a way to control the flow of information between layers.
mHC addresses these limitations by constraining the residual mapping onto a specific manifold: the set of doubly stochastic matrices (the Birkhoff polytope). This constraint ensures that the spectral norm of the mapping matrix is bounded by 1, making the residual transformation non-expansive. This increases numerical stability during both the forward pass and backpropagation.
The parameters of the three linear mappings in mHC (input, residual, and output) are dynamically generated from the input itself. They are decomposed into a dynamic (input-dependent) component and a static (input-independent) component. The residual mapping is projected onto the doubly stochastic manifold using the Sinkhorn-Knopp algorithm, which ensures that it has the desired stability properties.
In practice, mHC improves training stability and enables deeper networks without the numerical issues that can plague standard residual connections. It also provides a complementary scaling axis with minimal computational overhead, as the expansion factor n_hc is typically much smaller than the hidden size.
The Muon Optimizer
DeepSeek-V4 replaces the traditional AdamW optimizer with Muon for the majority of modules. Muon is a relatively new optimizer that has been shown to deliver faster convergence and greater training stability, particularly for large-scale models.
The key innovation in Muon is that it does not use element-wise second-moment estimation, which is the core of AdamW. Instead, it accumulates momentum and then performs orthogonalization on the momentum matrix using Newton-Schulz iterations. The orthogonalized update is then applied to the parameters.
For DeepSeek-V4, the Muon optimizer is configured with hybrid Newton-Schulz iterations. The first 8 iterations use aggressive coefficients to drive rapid convergence, while the final 2 iterations use stabilizing coefficients to precisely set the singular values to 1. This hybrid approach combines fast convergence with stability.
The Muon optimizer is not used for all modules. The embedding module, the prediction head, the static biases and gating factors of mHC, and the weights of all RMSNorm modules are still updated with AdamW. Everything else uses Muon.
One of the challenges with Muon is that it requires the full gradient matrix to compute parameter updates. This conflicts with the Zero Redundancy Optimizer (ZeRO), which is designed for element-wise optimizers like AdamW. DeepSeek addresses this with a hybrid ZeRO strategy: for dense parameters, they limit the size of ZeRO parallelism and use a knapsack algorithm to assign parameter matrices to ranks. For MoE parameters, they optimize each expert independently and pad the flattened vector to ensure even distribution across ranks.
The result is a training setup that is both efficient and stable, enabling the training of trillion-parameter models without the instability that often plagues large-scale training.
Inherited Design: DeepSeekMoE and Multi-Token Prediction
DeepSeek-V4 retains several design elements from its predecessors. The DeepSeekMoE framework, which uses fine-grained routed experts and shared experts for Feed-Forward Networks, is carried forward. However, there are some changes: the activation function for affinity scores is changed from Sigmoid to Sqrt(Softplus), and the constraint on the number of routing target nodes is removed.
The Multi-Token Prediction (MTP) strategy, which was validated in DeepSeek-V3, is also retained without modification. MTP trains the model to predict multiple future tokens simultaneously, which improves sample efficiency and enables faster convergence.
Chapter 3: The Infrastructure – Training and Inference at Scale
Fine-Grained Expert Parallelism
Training a 1.6 trillion parameter model is not easy. The Mixture-of-Experts (MoE) architecture helps by only activating a subset of parameters for each token, but it also introduces significant communication overhead. Expert Parallelism (EP) distributes experts across different GPUs, but this requires complex inter-node communication.
DeepSeek's solution is a fine-grained EP scheme that fuses communication and computation into a single pipelined kernel. The key insight is that communication latency can be hidden beneath computation. In each MoE layer, there are two communication-bound stages (Dispatch and Combine) and two computation-bound stages (Linear-1 and Linear-2). By overlapping these stages, the total time of communication is less than the total time of computation, meaning that communication does not become the bottleneck.
The scheme splits experts into waves. Each wave consists of a small portion of experts. As soon as all experts within a wave have completed their communication, computation can commence without waiting for other experts. In steady state, computation of the current wave, token transfer for the next wave, and result sending of completed experts all proceed concurrently.
This fine-grained approach achieves significant speedups: 1.50 to 1.73 times for general inference workloads, and up to 1.96 times for latency-sensitive scenarios like RL rollouts and high-speed agent serving.
TileLang: Flexible and Efficient Kernel Development
DeepSeek's elaborate model architecture would have resulted in hundreds of fine-grained Torch ATen operators. Instead, they adopted TileLang, a Domain-Specific Language (DSL) that balances development productivity with runtime efficiency.
TileLang allows developers to write high-performance kernels with minimal effort. It provides a unified codebase that supports rapid prototyping, large-scale training, and production deployment. The key features include:
Host Codegen: Moves most host-side logic into generated host code, reducing CPU-side orchestration overhead from tens or hundreds of microseconds to less than one microsecond per invocation.
SMT-Solver-Assisted Formal Integer Analysis: Integrates the Z3 SMT solver for formal analysis of integer expressions in tensor programs, enabling advanced optimizations like vectorization over variable tensor shapes.
Numerical Precision and Bitwise Reproducibility: Provides IEEE-compliant intrinsics with explicit rounding modes and enables bitwise reproducibility for validating kernels against hand-written CUDA baselines.
Batch-Invariant and Deterministic Kernels
Reproducibility is critical for large-scale training. DeepSeek has implemented end-to-end, bitwise batch-invariant and deterministic kernels with minimal performance overhead.
Batch Invariance ensures that the output of any given token remains bitwise identical regardless of its position within a batch. This is achieved through a dual-kernel strategy for attention: one kernel computes the attention output for an entire sequence within a single SM (ensuring high throughput for fully occupied waves), while a second kernel uses multiple SMs for a single sequence to minimize the latency of the final partially-filled wave. The two kernels are carefully designed to produce bitwise identical results.
Determinism is achieved by addressing the sources of non-determinism in training. For attention backward, separate accumulation buffers are allocated for each SM, followed by a global deterministic summation. For MoE backward, a token order pre-processing mechanism combined with buffer isolation across multiple ranks ensures determinism. For matrix multiplication in mHC, the split-k algorithm is modified to output each split part separately and perform a deterministic reduction in a subsequent kernel.
Training Framework Optimizations
The training framework builds on the infrastructure developed for DeepSeek-V3, with several key innovations.
Efficient Implementation of Muon: The Muon optimizer requires the full gradient matrix, which conflicts with ZeRO. DeepSeek uses a hybrid ZeRO strategy: for dense parameters, they limit the maximum size of ZeRO parallelism and assign parameter matrices to ranks using a knapsack algorithm. For MoE parameters, they optimize each expert independently and pad the flattened vector to ensure even distribution. The Newton-Schulz iterations are computed with BF16 matrix multiplications, and gradients are quantized to BF16 for communication, reducing communication volume by half.
Cost-Effective mHC: mHC increases activation memory consumption and communication volume. DeepSeek mitigates this with fused kernels, a recomputation strategy that selectively checkpoints intermediate tensors, and adjustments to the DualPipe 1F1B overlapping scheme. These optimizations constrain the wall-time overhead of mHC to only 6.7% of the overlapped 1F1B pipeline stage.
Contextual Parallelism for Long-Context Attention: Conventional Context Parallelism (CP) partitions the sequence dimension, but this creates challenges for compressed attention. DeepSeek uses a two-stage communication approach: first, each rank sends its last m uncompressed KV entries to the next rank, which compresses them together with its local entries. Second, an all-gather operation collects the locally compressed KV entries and reorganizes them into the full set.
Extended Automatic Differentiation: DeepSeek implemented a tensor-level activation checkpointing mechanism with automatic differentiation support. Developers can selectively annotate individual tensors for checkpointing and recomputation, and the framework automatically identifies the minimal subgraph required for recomputation. This provides fine-grained control without sacrificing programming efficiency.
Inference Framework
The inference framework includes several innovations for efficient KV cache management and on-disk storage.
Heterogeneous KV Cache Structure: The hybrid attention mechanism introduces multiple types of KV entries with different sizes and update rules. DeepSeek organizes the KV cache into two primary components: a classical KV cache for CSA/HCA, and a state cache for Sliding Window Attention (SWA) and uncompressed tail tokens. The state cache uses a fixed-size pool, while the classical KV cache allocates multiple blocks per request, each covering lcm(m, m') original tokens.
On-Disk KV Cache Storage: For shared-prefix requests, DeepSeek stores compressed KV entries on disk. For SWA KV entries, which are much larger, they implement three strategies: Full SWA Caching (stores everything), Periodic Checkpointing (checkpoints every p tokens), and Zero SWA Caching (stores nothing). The choice of strategy depends on the deployment scenario and the desired tradeoff between storage and computation.
Chapter 4: Pre-Training – Building the Foundation
Data Construction
DeepSeek-V4 is pre-trained on more than 32 trillion tokens of diverse, high-quality data. The corpus includes mathematical content, code, web pages, long documents, and other high-quality categories.
The data construction pipeline builds on the pre-training data of DeepSeek-V3, with several refinements. For web-sourced data, filtering strategies are implemented to remove batch-generated and templated content, mitigating the risk of model collapse. Mathematical and programming corpora remain core components, with additional agentic data incorporated during mid-training. For multilingual data, a larger corpus is built to improve capture of long-tail knowledge across different cultures. Special emphasis is placed on long-document data, prioritizing scientific papers, technical reports, and other materials with unique academic value.
The tokenizer remains the same as DeepSeek-V3, with a vocabulary size of 128K. Token-splitting and Fill-in-Middle (FIM) strategies are inherited, and documents from different sources are packed into appropriate sequences to minimize truncation. A new addition is sample-level attention masking during pre-training.
Training Setups
DeepSeek-V4-Flash has 43 layers and a hidden dimension of 4096. It uses CSA with compression rate m=4, 64 indexer query heads, and attention top-k of 512. HCA uses compression rate m'=128. Both use 64 query heads with a head dimension of 512. The sliding window size is 128. Each MoE layer has 1 shared expert and 256 routed experts, with 6 experts activated per token. Total parameters: 284B, activated: 13B.
DeepSeek-V4-Pro has 61 layers and a hidden dimension of 7168. CSA uses compression rate m=4, 64 indexer query heads, and attention top-k of 1024. HCA uses compression rate m'=128. Both use 128 query heads with a head dimension of 512. The sliding window size is 128. Each MoE layer has 1 shared expert and 384 routed experts, with 6 experts activated per token. Total parameters: 1.6T, activated: 49B.
Both models use Muon for most parameters and AdamW for embeddings, prediction heads, RMSNorm weights, and mHC biases. DeepSeek-V4-Flash is trained on 32T tokens with a peak learning rate of 2.7e-4, while DeepSeek-V4-Pro is trained on 33T tokens with a peak learning rate of 2.0e-4. The sequence length is gradually extended from 4K to 16K, 64K, and 1M. Dense attention is used for the first 1T tokens, after which sparse attention is introduced.
Mitigating Training Instability
Training trillion-parameter MoE models presents significant stability challenges. DeepSeek encountered loss spikes during training and developed two techniques to address them.
Anticipatory Routing: This technique decouples the synchronous updates of the backbone network and the routing network. At step t, the current network parameters are used for feature computation, but routing indices are computed using historical parameters from step t - Δt. This breaks the vicious cycle induced by routing and improves training stability. The overhead of Anticipatory Routing is bounded to approximately 20%, and it is only activated when a loss spike occurs.
SwiGLU Clamping: Empirically, DeepSeek found that clamping the linear component of SwiGLU to the range of [-10, 10] and capping the upper bound of the gate component at 10 effectively eliminates outliers and stabilizes training without compromising performance.
Evaluation of Base Models
The base models are evaluated across four dimensions: world knowledge, language understanding and reasoning, coding and mathematics, and long-context processing.
DeepSeek-V4-Flash-Base, despite having only 13B activated parameters, outperforms DeepSeek-V3.2-Base (37B activated) across a majority of benchmarks. This demonstrates that architectural improvements, refined data quality, and training optimizations yield superior performance even with a more compact parameter budget.
DeepSeek-V4-Pro-Base sets new performance highs among DeepSeek base models. It dramatically improves knowledge-intensive tasks, advances long-context understanding, and exceeds previous models on most reasoning and code benchmarks. The results confirm DeepSeek-V4-Pro-Base as the strongest foundation model in the DeepSeek series.
Chapter 5: Post-Training – From Base to Capable Assistant
The Post-Training Pipeline
The post-training pipeline of DeepSeek-V4 features a significant departure from DeepSeek-V3.2. The mixed Reinforcement Learning (RL) stage is completely replaced by On-Policy Distillation (OPD).
The pipeline consists of two stages:
Stage 1: Specialist Training. For each target domain (mathematics, coding, agent, instruction following), a separate expert model is trained independently. The base model first undergoes Supervised Fine-Tuning (SFT) on high-quality, domain-specific data. Then, Reinforcement Learning (RL) is applied using Group Relative Policy Optimization (GRPO), which further optimizes the model for domain-aligned behaviors guided by reward models tailored to specific success criteria. This yields a diverse set of specialized experts, each excelling in its respective field.
Generative Reward Models
For hard-to-verify tasks, DeepSeek uses a Generative Reward Model (GRM) instead of traditional scalar-based reward models. The GRM is optimized directly with RL, unifying the model's evaluative and generative capabilities. This approach achieves superior performance with only a minimal set of diverse human annotations, as the model leverages its own logic to generalize across complex tasks.
Reasoning Modes
DeepSeek-V4-Pro and DeepSeek-V4-Flash both support three reasoning effort modes:
| Mode | Characteristics | Response Format |
|---|---|---|
| Non-think | Fast, intuitive responses | </think> summary |
| Think High | Conscious logical analysis, slower but more accurate | <think> thinking </think> summary |
| Think Max | Push reasoning to its fullest extent | Special system prompt + <think> thinking </think> summary |
For the "Think Max" mode, a specific instruction is prepended to the system prompt: "Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem..."
Tool-Call Schema and Quick Instruction
DeepSeek-V4 introduces a new tool-call schema that employs a special "DSML" token and an XML-based format for tool invocations. The XML format mitigates escaping failures and reduces tool-call errors.
For auxiliary tasks (determining whether to trigger a web search, intent recognition, etc.), DeepSeek introduces Quick Instruction. Instead of using a separate small model, they append dedicated special tokens directly to the input sequence, where each token corresponds to a specific auxiliary task. By reusing the already-computed KV cache, this approach completely avoids redundant prefilling and reduces user-perceived time-to-first-token.
Interleaved Thinking
DeepSeek-V4 improves thinking management for agentic environments:
Tool-Calling Scenarios: All reasoning content is fully preserved throughout the entire conversation, including across user message boundaries. This allows the model to maintain a coherent, cumulative chain of thought over long-horizon agent tasks.
General Conversational Scenarios: Reasoning content from previous turns is discarded when a new user message arrives, keeping the context concise.
Infrastructure for Post-Training
Several infrastructure innovations support efficient post-training:
FP4 Quantization-Aware Training: FP4 quantization is applied to MoE expert weights and the Query-Key path in the indexer of CSA. This achieves a 2x speedup for the top-k selector while preserving a 99.7% recall rate. For MoE expert weights, FP32 master weights are quantized to FP4 and then dequantized back to FP8 for computation. This is lossless because FP8 has a larger dynamic range than FP4.
Efficient Teacher Scheduling for Full-Vocabulary OPD: All teacher weights are offloaded to distributed storage and loaded on demand with ZeRO-like parameter sharding. Last-layer teacher hidden states are cached in a centralized buffer, and the full logits are reconstructed on the fly by passing the cached states through the prediction head. Training samples are ordered by teacher index to ensure that each teacher head is loaded only once per mini-batch.
Preemptible and Fault-Tolerant Rollout Service: A token-granular Write-Ahead Log (WAL) is implemented for each generation request. When a new token is generated, it is immediately appended to the WAL. During preemption, the KV cache is saved. Upon resumption, the persisted WALs and saved KV cache are used to continue decoding. This is mathematically correct, unlike regenerating from scratch which introduces length bias.
Scaling RL Framework for Million-Token Context: Rollout data is decomposed into lightweight metadata and heavy per-token fields. Metadata is loaded for global shuffling, while heavy per-token fields are loaded via a shared-memory data loader. This substantially reduces both CPU and GPU memory pressure.
Sandbox Infrastructure for Agentic AI: DeepSeek Elastic Compute (DSeC) is a production-grade sandbox platform that manages hundreds of thousands of concurrent sandbox instances. It provides four execution substrates (Function Call, Container, microVM, fullVM) behind a unified interface, fast image loading via layered storage, density optimizations, and trajectory logging for preemption-safe resumption.
Chapter 6: Evaluation Results – How Good Is DeepSeek-V4?
Base Model Performance
DeepSeek-V4-Flash-Base, with only 13B activated parameters, outperforms DeepSeek-V3.2-Base (37B activated) across a wide array of benchmarks. This is particularly impressive in world knowledge tasks and challenging long-context scenarios.
DeepSeek-V4-Pro-Base demonstrates a further, decisive leap in capability, establishing near-universal dominance over both predecessors. It dramatically improves knowledge-intensive evaluations and advances long-context understanding.
Instruct Model Performance
DeepSeek-V4-Pro-Max, the maximum reasoning effort mode, significantly advances the knowledge capabilities of open-source models. Key results include:
Knowledge:
SimpleQA-Verified: 57.9% (vs. Gemini-3.1-Pro's 75.6%, but significantly ahead of other open models)
Chinese-SimpleQA: 84.4%
MMLU-Pro: 87.5%
Reasoning:
LiveCodeBench: 93.5% (top performance)
Codeforces: 3206 rating (23rd among human candidates)
GPQA Diamond: 90.1%
Agentic:
SWE Verified: 80.6%
Terminal Bench 2.0: 67.9%
Long Context:
MRCR 1M: 83.5%
CorpusQA 1M: 62.0%
DeepSeek-V4-Flash-Max
The smaller model achieves comparable reasoning performance to the Pro version when given a larger thinking budget, though it falls slightly behind on pure knowledge tasks and the most complex agentic workflows.
Formal Reasoning
DeepSeek-V4 demonstrates strong performance on formal mathematical tasks under both agentic and compute-intensive settings. Under an agentic setup, it achieves state-of-the-art results. With a more compute-intensive pipeline, performance further improves, matching the best known results under this setting.
Real-World Performance
Chinese Writing: DeepSeek-V4-Pro outperforms Gemini-3.1-Pro with a 62.7% win rate versus 34.1% on functional writing tasks. On creative writing, it achieves 60.0% in instruction following and 77.5% in writing quality. However, Claude Opus 4.5 retains an advantage on the most challenging prompts.
Search: Agentic search consistently outperforms Retrieval-Augmented Search (RAG), particularly on complex tasks. The agentic search is only marginally more expensive than standard RAG.
White-Collar Tasks: DeepSeek-V4-Pro-Max outperforms Opus-4.6-Max on diverse Chinese white-collar tasks with a 63% non-loss rate. It excels in Task Completion and Content Quality, proactively anticipating implicit user intents and delivering in-depth, coherent narratives.
Code Agent: DeepSeek-V4-Pro significantly outperforms Claude Sonnet 4.5 and approaches the level of Claude Opus 4.5 on real R&D coding tasks.
Chapter 7: DSpark – The Speculative Decoding Accelerator
One of the most important additions to the DeepSeek-V4 ecosystem is DSpark, a speculative decoding framework that dramatically improves inference speed without sacrificing output quality. This chapter provides a comprehensive analysis of DSpark, its architecture, performance characteristics, and deployment considerations.
What Is DSpark?
Speculative decoding is a technique that speeds up autoregressive generation by using a lightweight draft model to predict multiple future tokens in parallel, which are then verified by the main model. DSpark is DeepSeek's implementation of this technique, optimized specifically for the DeepSeek-V4 architecture.
The Need for Speculative Decoding
Autoregressive generation in large language models is inherently sequential. Each token must be generated one at a time, with each step depending on the previous one. This creates a fundamental bottleneck: the generation speed is limited by the latency of a single forward pass through the model.
For large models like DeepSeek-V4-Pro with 49B activated parameters, this latency can be significant. In high-throughput serving scenarios, the cost of generating each token individually becomes prohibitive, especially for applications requiring long outputs or real-time interaction.
Speculative decoding addresses this bottleneck by introducing a "draft-then-verify" paradigm:
Draft Phase: A lightweight draft model generates a sequence of future tokens quickly.
Verification Phase: The main model processes the draft sequence in parallel, verifying which tokens are correct.
Acceptance: Correct tokens are accepted, and the process repeats from the first incorrect token.
This approach can achieve significant speedups because the verification phase processes multiple tokens in a single forward pass, amortizing the cost of the main model over multiple generated tokens.
DSpark's Innovations
DSpark builds on the foundation of speculative decoding with several key innovations:
1. Semi-Autoregressive Draft Generation
Traditional speculative decoding uses either a parallel draft model (fast but prone to quality degradation in later tokens) or a sequential draft model (accurate but slower). DSpark introduces a semi-autoregressive approach that strikes a balance between speed and quality.
The draft model generates batches of tokens in parallel while maintaining lightweight sequential dependencies between batches. This hybrid strategy achieves the speed of parallel generation while preserving the quality of autoregressive generation.
2. Confidence-Scheduled Verification
In high-load serving scenarios, verifying low-confidence draft tokens can waste valuable GPU cycles. DSpark introduces a confidence-scheduled verification mechanism that dynamically adjusts verification behavior based on real-time system load:
When the system is idle: DSpark verifies longer draft sequences, maximizing the speedup.
When the system is busy: DSpark truncates low-confidence tokens, prioritizing throughput.
This load-adaptive approach ensures stable performance across varying load conditions.
3. Hardware-Aware Scheduling
DSpark includes a hardware-aware scheduler that optimizes the draft-verify pipeline for specific hardware configurations. The scheduler accounts for:
GPU memory bandwidth
Compute capacity
Interconnect latency (for multi-GPU setups)
This ensures that DSpark achieves near-optimal performance across diverse hardware environments.
Performance Results
DSpark has been deployed in DeepSeek's production systems and evaluated under real user traffic. The performance gains are substantial:
| Model | Generation Speed Improvement |
|---|---|
| DeepSeek-V4-Flash | 60% to 85% |
| DeepSeek-V4-Pro | 57% to 78% |
These speedups are measured at the same throughput level, meaning that DSpark enables the model to generate outputs significantly faster while serving the same number of users.
In offline benchmark evaluations, DSpark achieves an average draft acceptance length 26.7% to 30.9% higher than Eagle3 and 16.3% to 18.4% higher than DFlash on Qwen3 series models. This means that each verification round accepts more tokens, leading to higher efficiency.
DeepSpec: The Open-Source Toolchain
DSpark is part of a broader open-source effort called DeepSpec, which provides a full-stack codebase for training and evaluating speculative decoding draft models. DeepSpec includes:
Data Preparation Tools: Scripts for downloading prompts, regenerating target answers, and building target caches.
Draft Model Implementations: Built-in support for DSpark, DFlash, and Eagle3 algorithms.
Training Code: Support for training draft models on 8-GPU nodes.
Evaluation Scripts: Coverage of GSM8K, MATH500, HumanEval, MBPP, LiveCodeBench, MT-Bench, and more.
DeepSpec is released under the MIT License, allowing developers to freely use, modify, and deploy it. Researchers can directly train their own speculative decoding draft models on DeepSpec, significantly lowering the deployment barrier.
Deployment with vLLM
Deploying DSpark is straightforward. In vLLM, the most popular open-source inference engine for LLMs, DSpark is enabled with a single flag:
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'vllm serve deepseek-ai/DeepSeek-V4-Pro-DSpark \ --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \ --data-parallel-size 4 --enable-expert-parallel \ --moe-backend deep_gemm_mega_moe \ --attention-config '{"use_fp4_indexer_cache": true}' \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'
For DeepSeek-V4-Flash-DSpark, the configuration is identical, simply substituting the model name.
Strategic Significance
DSpark represents a strategic shift in the AI industry. The release of DSpark, following DeepSeek's completion of a 50 billion RMB funding round, signals that the competition in AI is extending beyond model capabilities to systems engineering and inference efficiency.
The reasoning is clear: model capability alone is insufficient for practical deployment. If inference is too slow or too expensive, even the most capable model will struggle to find adoption. By optimizing inference efficiency through techniques like speculative decoding, DeepSeek is making its models more accessible and cost-effective for real-world applications.
This focus on efficiency is particularly important for long-context scenarios. In the 1-million-token context setting, even with the efficient CSA/HCA architecture, the cost of generation is substantial. DSpark reduces this cost further, making million-token contexts practical for routine use.
Chapter 8: Limitations and Future Directions
Current Limitations
DeepSeek-V4's bold architectural design comes with some complexity. The hybrid attention mechanism, while efficient, is relatively complex compared to simpler alternatives. Many components and tricks from previous models were retained to minimize risk, making the architecture more complex than it needs to be.
The techniques used to mitigate training instability, Anticipatory Routing and SwiGLU Clamping, are empirically effective but not fully understood theoretically. Their underlying principles remain an open question.
The model's performance on formatting aesthetics and instruction following in some scenarios still lags behind the best proprietary models. It occasionally overlooks specific formatting constraints and is less proficient at condensing extensive text inputs into succinct summaries.
Future Directions
DeepSeek plans to carry out more comprehensive investigations to distill the architecture down to its most essential designs, making it more elegant without sacrificing performance.
They will actively study foundational problems on training stability and strengthen internal metric monitoring, aiming for a more principled and predictive approach to stable large-scale training.
Beyond MoE and sparse attention, DeepSeek will explore model sparsity along new dimensions, such as more sparse embedding modules, to further improve computational and memory efficiency.
They will continuously investigate low-latency architectures and system techniques to make long-context deployment and interaction more responsive.
DeepSeek recognizes the importance of long-horizon, multi-round agentic tasks and will continue to iterate and explore in this direction.
They are also working on incorporating multimodal capabilities into their models.
Finally, they are committed to developing better data curation and synthesis strategies to consistently enhance model intelligence, robustness, and practical usability.
Chapter 9: Frequently Asked Questions
Conclusion
DeepSeek-V4 represents a significant leap forward in the efficiency and capability of large language models. By introducing a hybrid attention architecture that combines Compressed Sparse Attention and Heavily Compressed Attention, DeepSeek has broken the quadratic bottleneck that has long constrained ultra-long-context processing.
The results speak for themselves. In the 1-million-token context setting, DeepSeek-V4-Pro requires only 27% of the single-token inference FLOPs and 10% of the KV cache compared with its predecessor. This is not a marginal improvement. It is a fundamental breakthrough that makes million-token contexts practical for real-world deployment.
But efficiency is only part of the story. DeepSeek-V4-Pro-Max delivers state-of-the-art performance on knowledge benchmarks, achieves top-tier results on coding tasks, and significantly bridges the gap with leading proprietary models on reasoning and agentic tasks. It is the best open-source model available today.
The innovations in DeepSeek-V4 extend beyond the attention mechanism. The Manifold-Constrained Hyper-Connections strengthen training stability, the Muon optimizer delivers faster convergence, and the fine-grained expert parallelism scheme enables efficient training at scale. The post-training pipeline, with its two-stage paradigm of specialist cultivation and on-policy distillation, produces a model that excels across diverse domains.
The addition of DSpark further enhances the practical value of DeepSeek-V4, delivering 60-85% faster generation on Flash and 57-78% faster on Pro. This makes million-token contexts not just technically feasible but economically practical for routine deployment.
The future of AI increasingly depends on the ability to process and reason over long contexts. Agentic workflows, massive cross-document analysis, and complex multi-step problem solving all require models that can maintain context over extended sequences. DeepSeek-V4 lays the foundation for these capabilities, enabling the next generation of test-time scaling and long-horizon reasoning.
As DeepSeek continues to iterate and explore new directions, we can expect even more impressive advances. The roadmap includes architectural distillation, multimodal capabilities, and deeper exploration of long-horizon agentic tasks. The era of million-token context intelligence has arrived.
I hope this guide has helped you understand the remarkable innovations in DeepSeek-V4 and DSpark. If you have any questions or thoughts, please feel free to share them. Thank you for reading.
https://u.pcloud.link/publink/show?code=kZqBBFJZgUd3aDJld5kbxcz3S8gzt8i1VGvV
ReplyDelete