Multimodal Agent MemorySubmit a paper

MovieChat

MovieChat: From Dense Token to Sparse Memory for Long Video Understanding

Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, et al. · 2023-07-31

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. short-term and long-term memory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    A bounded short-term buffer stores frame-level visual feature tokens and periodically transfers them for consolidation.Full paper: 2.3. Memory Models in Vision Tasks · PDF p. 3 · Figure 2; Equation 2
    Adjacent visual tokens are iteratively merged by similarity and the sparse result is appended to long-term memory.Full paper: 3.4. Long-term Memory · PDF p. 4 · Algorithm 1
    Global and breakpoint modes activate the stored visual representation through projection into the language model; no source-frame lookup is defined.Full-paper method / memory representation and retrieval · PDF p. 5 · Equation 4

How memory changes (T2 operations)

  • Consolidation · Merging & deduplication

    A fixed, parameter-free consolidation rule repeatedly merges the most similar adjacent frame tokens. For MovieChat, this specifically supports merging or deduplication of overlapping entries of short-term and long-term memory.

  • Reorganization · Memory-level migration (short- to long-term)

    Compressed short-term tokens are appended to long-term memory and initialize the next short-term segment, explicitly migrating state between tiers. For MovieChat, this specifically supports migration between memory tiers of short-term and long-term memory.

Related systems