DeepSeek V4.1 Flash
Scroll sideways to see the whole figure.
Figure | Overall architecture of DeepSeek-V4.1-Flash. The 40-layer network is divided into a causal encoder and a decoder, each with 20 layers. All feed-forward layers use standard DeepSeekMoE. The first two encoder layers use sliding window attention (SWA); the rest use Compressed Sparse Attention 2 (CSA2), with CSA2(ratio, mode) specifying the compression ratio and mode. The model also uses Single-Pass mHC, Engram, DSpark, and a Hierarchical Sparse Indexer.
Structural changes
Asymmetric encoder–decoder (CED)
Both stacks are causal: every layer attends only to earlier tokens. The split concerns what each half produces, not which direction it can see. The lower 20 layers produce HL/2, and the upper 20 layers take their global KV from it rather than computing their own. Generated tokens pass through both halves; only prompt tokens can stop after the lower one.
The asymmetry is in reading the prompt. Decoder global KV is projected from the encoder's final hidden state, so prompt tokens only need the 20 encoder layers, plus a 128-token decoder replay. The model activates 8B parameters per token in prefill and 16B in decode, and prefill compute is nearly halved.
CSA2 with cross-layer reuse
| Mode | Main KV / indexer K | Top-K indices |
|---|---|---|
| Full | Computes its own | Computes fresh |
| Reindex | Reuses | Own indexer Q, fresh Top-K |
| Reuse | Reuses | Reuses |
Every mode still computes its own main Q and SWA KV. The real pattern is: encoder SWA, SWA, then [Full + 5 Reuse] ×3, with no Reindex. The decoder is Full + 3 Reuse, then [Reindex + 3 Reuse] ×4. The report gives two reasons: shared KV reduces cache storage, and reused Top-K avoids indexer compute. V4.1 also drops V4's CSA–HCA hybrid for pure CSA2.
Engram conditional memory
The report integrates Engram, "the conditional memory module introduced in our previous work", to decouple memorization from computation. The report does not say Engram was in an earlier model and later removed.
Engram hashes the recent 2-, 3- and 4-grams into large tables (196B parameters, two modules, at layers 1 and 14) and gates the retrieved embeddings into the network. Addresses depend only on the input tokens, so lookups can be prefetched from host memory.
- MoE
- Each token uses 6 of 384 routed experts plus 1 shared expert.
- Sparse attention
- Each query reads its Top-512 main-KV entries plus a 128-token SWA window, not the whole context.
- CSA2 reuse
- Only 4 layers store global KV and 8 of the 38 CSA2 layers run an indexer derived from §4.2.1.
- Hierarchical indexer
- Later decoder indexers score at most 16,384 candidates, whatever the context length.
- Engram
- Hashed table lookup, decoupling memorization from computation.
- CED
- Prompt tokens skip the decoder's full forward pass.
All 40 layers
The figure's ×N groups unrolled. Select a layer for its details.
How Full, Reuse and Reindex pick tokens
A 16-token sequence with the query at position 15. Sizes are shrunk so everything fits on screen. The toy uses Top-3 instead of 512, a 4-token SWA window instead of 128, blocks of 2 instead of 8, and a 4-block pool instead of up to 2,048. The scores are made-up illustrations.