Multimodal Agent MemorySubmit a paper

MovieChat+

MovieChat+: Question-aware Sparse Memory for Long Video Question Answering

Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, Gaoang Wang · 2024-04-26

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. question-aware sparse visual token memory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search
    4. Source evidenceRetained but not accessible to the agent
    Question-aware sparse memory compresses long video tokens while preserving query-relevant content.Full paper: Memory · PDF p. 4 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Merging & deduplication

    Redundant visual tokens are merged under the fixed Atkinson-Shiffrin-inspired policy. For MovieChat+, this specifically supports merging or deduplication of overlapping entries of question-aware sparse visual token memory.

  • Consolidation · Episodic / hierarchical summarization

    Long streams are consolidated into sparse memory. For MovieChat+, this specifically supports episodic or hierarchical summarization of question-aware sparse visual token memory.

Related systems