Multimodal Agent MemorySubmit a paper

WnW (Waxing-and-Waning KV Cache)

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

Yiming Yao, Chenyang Lyu, Xuanfan Ni, Longyue Wang, Weihua Luo, Yazheng Yang, et al. · 2026-08-24

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Audio & speech
Also: text
Representation
Internalized latent & parametric

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. chunked audio KV cache split across a GPU-resident working set and a CPU-resident recallable backing store

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval, Latent / parametric activation
    4. Source evidenceRetained but not accessible to the agent
    Audio key-value states for a ten-minute clip (7,500-15,000 KV positions at 12.5-25 audio tokens/s) are kept as an operative latent memory. Offline calibration classifies attention heads into ~5 anchor heads that retain the whole audio KV on GPU, tidal heads that keep part on GPU and offload the rest to CPU with recall, and fixed heads that keep only a static prefill-selected subset. Text-token KV is kept in full; only audio KV is compressed.Method: head triage and two-tier cache · Head-classification / cache-layout figure
    The store is organised into 4-second chunks with 2-second stride, i.e. indexed by audio time. At each decode step the anchor heads' softmax weights over audio positions are aggregated into a per-position importance score, chunks are ranked, and the top-k (k=3) chunks are recalled from CPU into GPU if absent. Retrieval is therefore attention-score-based over temporally indexed chunks, and the recalled KV is consumed directly by attention rather than being decoded back to any surface form.Method: decode-time chunk scoring and recall · Retention equation retention_i = min(s_i * lambda, 1.0); chunk-scoring algorithm

How memory changes (T2 operations)

  • Reorganization · Memory-level migration (short- to long-term)

    During generation, chunks selected by the anchor-head score are migrated from the CPU tier back into GPU residency and unselected chunks are demoted to CPU only. The GPU/CPU placement of the audio KV memory therefore changes continuously at inference time under a fixed top-k/decay rule.

  • Forgetting · Discrete eviction

    A chunk that is not selected for delta = 3 consecutive decode steps is evicted from the GPU working set (the CPU copy persists for tidal heads, while fixed heads have no recovery path, making eviction there permanent). Eviction is a deterministic inference-time deletion under a fixed threshold.

Related systems