Multimodal Agent MemorySubmit a paper

MA-LMM

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, et al. · 2024-04-24

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Image, text
Representation
Hybrid
The dual bank jointly activates modality-native frame features and persistent learned query states in the Q-Former.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. visual memory bank

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Sequentially encoded frame features are cached as keys and values so the current Q-Former queries can attend to visual history.Full paper: 3. Method · PDF p. 3 · Figure 2

  2. query memory bank

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inHierarchical multi-tier store
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Processed query representations from previous time steps persist separately and enter later Q-Former self-attention.Full paper: 3. Method · PDF p. 3 · Figure 2

How memory changes (T2 operations)

  • Update · Learned update module

    A fixed trained Q-Former generates and appends the query-state memory as frames arrive. For MA-LMM, this specifically supports a trained but frozen update module of visual memory bank.

  • Consolidation · Merging & deduplication

    When a bank exceeds its bound, the fixed compression policy finds maximally similar adjacent entries and averages them. For MA-LMM, this specifically supports merging or deduplication of overlapping entries of visual memory bank.

Related systems