DeepSeek Unplugged Its Memory Module. TriviaQA Kept 29 Percent.

The paper is Conditional Memory via Scalable Lookup, from DeepSeek-AI and Peking University. Version 1 went up on January 12, 2026. Version 2, the one this issue reads, went up on July 12, 2026. The last name on the author line is Wenfeng Liang, DeepSeek's founder. Code is at github.com/deepseek-ai/Engram.

The claim: Mixture-of-Experts made computation conditional. Nothing in a Transformer made memory conditional. Every time a model recognises a fixed phrase, a name, an idiom, a code identifier, it rebuilds that phrase through attention and feed-forward layers, which the authors call simulating a lookup with compute. Engram adds the lookup. Recent tokens become an address. The address hits a table. The row comes back in constant time.

Where it landed: the DeepSeek-V4.1-Flash model card lists a 552B backbone and a separate 196B Engram memory, with 8B activated at prefill and 16B at decode. The weights on Hugging Face total about 763B. So the number you quote for this model depends on whether you count the table.

Six steps, all deterministic until the gate.

One consequence of step 2 is worth holding onto. The address is the token sequence itself, not the hidden state. The model knows every Engram row it will need before the forward pass starts. That single property is what listing 07 is built on.

Fix total parameters and FLOPs per token. Now decide how much of the inactive budget goes to routed experts and how much to Engram rows. The paper calls the expert share rho and sweeps it at two budgets, 2e20 and 6e20 FLOPs, keeping total-to-active at roughly 10 to 1.

The gap at the optimum is 0.0139 in validation loss. Small in absolute terms, but it costs nothing in FLOPs, and pure MoE sits on the wrong side of it. A second experiment holds a 3B backbone fixed and grows the table from 258 thousand slots to 10 million (about 13B added parameters). Loss falls as a straight line in log space across the whole range. The table is a scaling knob that costs no compute.

MoE-27B has 2 shared plus 72 routed experts, top-6. Engram-27B cuts routed experts to 55 and spends the freed 5.7B on the table, so rho lands at 74.3 percent, inside the band from listing 03. Both train on 262B tokens. Engram-40B keeps the same backbone and grows the table to 18.5B.

Two things the table says that the abstract does not. The largest single gain is on Chinese classical poetry matching, 7.5 points, which is the purest memorised-phrase task in the suite, and the smallest is PopQA, 0.2 points, the long-tail fact benchmark where a lookup table should shine. A 3-gram over compressed tokens is a phrase memory, not a fact store. Second, the 40B model with three times the table loses to the 27B on HumanEval, MBPP and MATH (red cells), and on C3, which is not shown. The authors call it under-training and note the loss gap was still widening at the end of the run. Hold that explanation loosely until someone trains it longer.

The authors zeroed the module's output at inference, backbone untouched, and re-ran the benchmarks. They warn the test is noisy because the network never trained without its table, so they report the two ends of the spectrum.

Read the two rows together. Cut the wire to the table and the model forgets most of what it knew, but still reads a passage put in front of it almost as well as before. The facts moved into the rows. The reasoning stayed in the backbone. That is a cleaner division of labour than anything an MoE router gives you, because a router's experts are not legible as "the knowledge" and "the logic".

Two interpretability probes point the same way. LogitLens shows the Engram model's early layers produce distributions closer to its final answer than the baseline's do. CKA alignment on named entities shows representations at layer 5 of Engram-27B line up with roughly layer 12 of MoE-27B. The authors' reading: the table does the first seven layers of phrase assembly, so the network behaves as if it were deeper. The paper's own example is the entity "Diana, Princess of Wales", which a standard model resolves one layer at a time from Wales, to a title, to the person, by layer 6.

Both 27B models got the same YaRN extension to 32k, 5,000 steps, 30B tokens. The twist is which Engram checkpoint goes in. The 46k-step checkpoint has the same pre-training loss (1.63) as the finished MoE at 50k, so any gap after extension belongs to the architecture, not to a stronger starting model.

The authors' explanation is that local dependencies go to the table, so attention has capacity left for global context. The table supports that on multi-query retrieval and variable tracking. It also shows two things the abstract skips: the baseline wins single-needle retrieval at 100.0 against 97.6, and common words extraction is a wreck for every model in the table, under 6 percent across the board. An early-stopped Engram at 41k steps, 82 percent of the baseline's compute, still matches the baseline's long-text perplexity and beats it on RULER.

Because the address is known before the forward pass, the rows can be fetched while the first block computes. The authors put a 100B-parameter Engram table in host DRAM, inserted it at block 2 of a dense model, and ran a nano-vLLM harness on an NVIDIA H800 with 512 sequences of 100 to 1,024 tokens.

A hundred billion parameters that never touch HBM, at a throughput cost under three percent. Traffic per step scales with rows activated, not table size, so the table can keep growing. The design section goes further: N-gram hits are Zipfian, so hot rows can sit in HBM, warm rows in DRAM, the long tail on NVMe. The benchmark above did none of that and sent every row over PCIe.

Two caveats. The test backbones are dense, chosen to avoid expert-parallel traffic, so a production MoE serving stack competes for the same PCIe and interconnect. And the latency window is one block wide because the table sits at layer 2, exactly where the modelling ablation wants it. The hardware wants it deep, the loss wants it shallow, and the paper picks shallow.

What shipped follows the design, not the benchmark. Unsloth's V4.1-Flash guide runs the 196B table from SSD and says the production model addresses N-grams of up to four tokens, one more than the paper's three. The community GGUF builds pass a flag that pins the Engram weight tensor to CPU RAM and loads it lazily. The paper's memory hierarchy is already a command-line option.

Parameter counts now need two columns. A 748B number for V4.1-Flash and a 552B number are both defensible. When you compare models, ask which column the table is in, and whether active parameters are quoted at prefill or decode. The paper's own metric for this is P sparse, the parameters you carry but do not compute, and Engram rows are the cheapest P sparse there is.

Local and edge serving math changes. Memory-class parameters do not need HBM. A model with a quarter of its weights in an N-gram table can run on a card that holds the backbone, with the table on the host or on disk, at a cost the paper measures in low single digits. Unsloth is already shipping that recipe for V4.1-Flash. Watch for the same split in the next open releases from other labs.

Knowledge might become addressable. This one is my inference, not the paper's claim. Facts collapsed to 29 percent when the table was unplugged, and every row has a deterministic address computed from tokens alone. That is the precondition for editing a model's knowledge by rewriting rows instead of fine-tuning weights, the thing ROME and MEMIT tried to do to feed-forward layers. The paper cites that literature and stops there. If anyone demonstrates row-level editing on an Engram model, that will be a bigger story than the benchmark table.