Multimodal Agent MemorySubmit a paper

Video-EM

Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding

Yun Wang, Long Zhang, Jingren Liu, Jiaqi Yan, Zhanjie Zhang, Jiahao Zheng, et al. · 2026-03-07

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Image, text
Representation
Hybrid
Structured event records remain grounded in the selected keyframes used by the downstream Video-LLM.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. query-grounded keyframes grouped into events

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Multi-grained matching localizes moments and event expansion preserves the selected visual frames as grounded evidence.Full-paper method / memory representation and retrieval · PDF p. 3

  2. dynamic scene narratives and relations with temporal indices

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Each event explicitly records when, where, what, involved entities, dynamic scene relations, and a high-level narrative.Full paper: Self-reflective Memory Refinement · PDF p. 7 · Figure 4

  3. grounded episodic narratives with when, where, what, and who cues

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records, Graph / relational store
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Selected events are written as textual episodic records containing temporal indices, scene narratives, and when/where/what/who cues linked to their keyframes.Full-paper method / memory representation and retrieval · PDF p. 3 · Figure 4

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    An LLM memory agent iteratively checks sufficiency and cross-event consistency and adjusts event granularity. For Video-EM, this specifically supports LLM-mediated memory editing of query-grounded keyframes grouped into events.

  • Consolidation · Merging & deduplication

    The reflection loop removes redundant evidence to produce a minimal sufficient event set. For Video-EM, this specifically supports merging or deduplication of overlapping entries of query-grounded keyframes grouped into events.

Related systems