Multimodal Agent MemorySubmit a paper

MM-Mem

From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents

Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, et al. · 2026-03-02

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Image, text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. fine-grained perceptual traces

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource directly available
    The Sensory Buffer retains verbatim traces.Full paper: 3. Method · PDF p. 3 · Figure 1

  2. episodic stream and symbolic schema

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    The pyramid distills sensory traces into episodes and gist.Full paper: 3. Method · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Update · Learned update module

    During operation, MM-Mem applies a trained but frozen update module to fine-grained perceptual traces; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Episodic / hierarchical summarization

    A fixed pyramid summarizes verbatim traces. For MM-Mem, this specifically supports episodic or hierarchical summarization of fine-grained perceptual traces.

  • Consolidation · Experience abstraction

    The information bottleneck abstracts task-relevant gist. For MM-Mem, this specifically supports abstraction of episodes into reusable experience of fine-grained perceptual traces.

  • Reorganization · Memory-level migration (short- to long-term)

    Information moves through sensory, episodic, and symbolic tiers. For MM-Mem, this specifically supports migration between memory tiers of fine-grained perceptual traces.

  • Consolidation · Merging & deduplication

    During operation, MM-Mem applies merging or deduplication of overlapping entries to fine-grained perceptual traces; the memory state changes while the governing update policy remains fixed.

Related systems