Multimodal Agent MemorySubmit a paper

video-SALMONN S

video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLM

Guangzhi Sun, Yixuan Li, Xiaodong Wu, Yudong Yang, Wei Li, Zejun Ma, et al. · 2026-02-03

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Audio & speech, text
Representation
Hybrid
The TTT fast-weight long-term memory, the recurrent multimodal memory-token bank, and the prompt-compressed modality-native KV cache are jointly supplied to the LLM when answering at a given point in the stream.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. TTT_MEM fast weights W_t that absorb streamed audio-visual chunks into long-term memory held in model parameters

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    The layer computes (Z_t, W_t) = TTT_MEM(X_t, X_{t-T}, W_{t-1}); the fast weights W_t are updated at streaming time by a gradient step W_t = W_{t-1} - eta * grad l(X_t, X_{t-T}; W_{t-1}) on a reconstruction loss (Eq. 2) plus a long-span prediction loss (Eq. 3) that 'encourages retention of information relevant across distant timestamps'. The abstract states the model 'continuously transforms short-term multimodal representations into long-term memory embedded in model parameters', so history persists only inside the weights and no source frames or audio are recoverable.Full paper: Sec. 3.1 Test-time training memory (TTT_MEM) · HTML Sec. 3.1 · Eq. 1-4

  2. fixed-size bank of N multimodal memory tokens Z~_t carried recurrently across streaming chunks

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceRetained but not accessible to the agent
    A fixed-size sequence of N memory tokens (16k default, up to 128k) persists across chunks; incoming tokens Z_t are merged with the previous bank Z~_{t-1} and the budget is held by discarding the most cosine-similar adjacent token pairs. This bank is a deliberate recurrent multimodal latent state supplied to the LLM at later chunks, supporting streams of over three hours at 11 FPS.Full paper: Sec. 3.1 · HTML Sec. 3.1 · Eq. 1; Figure 2/3 (architecture)

  3. prompt-dependent compressed KV-cache used for modality-aware memory reading

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved bySimilarity search
    4. Source evidenceSource not retained
    At read time the retained per-layer key/value cache is compressed in a prompt-dependent way: the top-M tokens per layer by attention score against the current question are kept with their relative positions preserved, so which cached audio-visual encodings reach the answer depends on the query.Full paper: Sec. 3.3 Modality-aware memory reading · HTML Sec. 3.3 · Sec. 3.3 description

How memory changes (T2 operations)

  • Update · Optimization-based update

    During streaming inference the fast-weight memory is revised by an explicit gradient-descent step per chunk on reconstruction and long-span prediction objectives; the memory content (weights) changes while the update rule, span T = 2 and learning rate eta remain fixed.

  • Consolidation · Merging & deduplication

    New chunk tokens are merged into the persistent memory-token bank and the bank is held at its budget by discarding the K highest-similarity adjacent token pairs, a fixed deduplication rule applied at streaming time.

  • Forgetting · Discrete eviction

    Prompt-dependent KV-cache compression drops all but the top-M attention-scored tokens per layer at inference time under a fixed selection rule.

Related systems