Multimodal Agent MemorySubmit a paper

HippoMM

HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding

Yueqian Lin, Jingyang Zhang, Qinsi Wang, Hancheng Ye, Yuzhe Fu, Yudong Liu, et al. · 2026-04-01

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: Audio & speech, Image, text
Representation
Hybrid
Abstract summaries, representative embeddings, and linked audiovisual episode details are bundled in each ThetaEvent.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. key visual and auditory context linked from ThetaEvents

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byMetadata / spatiotemporal retrieval, Similarity search
    4. Source evidenceSource linked & recoverable
    Each ThetaEvent retains representative cross-modal embeddings and references to key visual, auditory, and temporal context in the detailed short-term segment.Full paper: 4.2. Hierarchical Memory Retrieval (Pattern Comple- · PDF p. 5

  2. semantic ThetaEvent summaries

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Consolidated audiovisual segments become concise summaries; confidence-gated retrieval searches summaries first and escalates to detailed temporal windows when needed.Full paper: 4.2. Hierarchical Memory Retrieval (Pattern Comple- · PDF p. 5 · Figure 2

  3. detailed audiovisual source spans linked from ThetaEvents

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Each ThetaEvent retains pointers to detailed audiovisual source spans, and confidence-gated coarse-to-fine retrieval can reopen the underlying temporal evidence.Full paper: 2.1. Hippocampal Memory Mechanisms · PDF p. 2 · Figure 2

Related systems