Multimodal Agent MemorySubmit a paper

DriveWAM Selective KV Memory

DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving

Chen Shi, Jinrui Xu, Shaoshuai Shi, Kehua Sheng, Bo Zhang, Li Jiang · 2026-05-27

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: actions, text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. bounded modality-aware KV memory pools for video latent tokens and action tokens carried across autoregressive chunks

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Video and action key/value tensors from the diffusion transformer are kept in two separate bounded pools (video budget 448 tokens, action budget 160 tokens) that persist across chunks of a long driving rollout. They are read back by ordinary self-attention when the next chunk is generated. Only KV tensors of VAE-encoded latents are retained; original frames are not recoverable from the pool.Full paper: Sec. 3.3 Selective KV Memory; Sec. 4.2 implementation · HTML Sec. 3.3 / 4.2 · Figure 3 (retained video tokens); Eq. 6-8; Table 5

How memory changes (T2 operations)

  • Update · Rule-based update

    After each generated chunk the memory pool is rewritten as H^{k+1}_m <- TopB(H^k_m) union deltaH^{k+1}_m, i.e. the new chunk's KV entries are appended to the persistent pool under a fixed rule during inference.

  • Forgetting · Discrete eviction

    Cached tokens are scored by s_j = lambda * relevance (attention mass from current queries) - (1-lambda) * redundancy (cosine similarity to other cached keys), with lambda = 0.07; tokens below the retention budget are discarded at inference time. Figure 3 shows repeated road/sky/building tokens being evicted while moving vehicles and lane geometry are retained; Table 5 reports 3.07 GB -> 0.25 GB over a 300 s rollout.

Related systems