Multimodal Agent MemorySubmit a paper

WorldMM

WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning

Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, Sung Ju Hwang · 2026-03-27

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Image, text
Representation
Hybrid
Graph-derived time ranges can directly index original frames, while feature search independently retrieves visual segments.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. timestamped frames and searchable segment features

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection, Flat sequential records
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    WorldMM stores segment features for text-to-video similarity search and a timestamp-to-frame index for direct visual inspection.Full paper: 3.2. Adaptive Memory Retrieval · PDF p. 4

  2. multi-scale event graphs and evolving semantic graph

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Hierarchical multi-tier store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Episodic knowledge graphs span multiple temporal scales; a semantic graph stores long-term relations, and a retrieval agent iteratively chooses memory type and query.Full paper: Memory · PDF p. 2 · Figure 2

How memory changes (T2 operations)

  • Revision · LLM-based conflict resolution

    Matched old and new triplets are sent to an LLM to identify outdated facts and revised or added facts. For WorldMM, this specifically supports LLM-based resolution of conflicting records of timestamped frames and searchable segment features.

  • Consolidation · Merging & deduplication

    Embedding similarity finds overlapping triplets before consolidation integrates new conceptual knowledge. For WorldMM, this specifically supports merging or deduplication of overlapping entries of timestamped frames and searchable segment features.

  • Consolidation · Experience abstraction

    Semantic triplets and recurring habits are extracted from video segments and consolidated into a reusable long-term semantic graph. For WorldMM, this specifically supports abstraction of episodes into reusable experience of timestamped frames and searchable segment features.

Related systems