Multimodal Agent MemorySubmit a paper

Embodied VideoAgent

Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding

Yue Fan, Xiaojian Ma, Rongpeng Su, Jun Guo, Rujie Wu, Xi Chen, et al. · 2025-01-09

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Video & streaming, Image, depth, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. object states, relations, 3D boxes, and visual/context features

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Vector-indexed collection
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceRetained but not accessible to the agent
    Each persistent object entry stores identity, state, relations, 3D bounding box, object visual feature, and contextual visual feature; tools query these fields.Full paper: 2.2. Memory and Tools of Embodied VideoAgent · PDF p. 3 · Figure 3

  2. segment captions and video features

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records, Vector-indexed collection
    3. Retrieved byMetadata / spatiotemporal retrieval, Similarity search
    4. Source evidenceRetained but not accessible to the agent
    A temporal table stores segment identifiers, captions, video features, and caption embeddings for tool-based temporal localization.Full paper: 2.2. Memory and Tools of Embodied VideoAgent · PDF p. 3

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    A VLM determines object-state and relation changes caused by observed actions and rewrites the corresponding persistent object entries. For Embodied VideoAgent, this specifically supports LLM-mediated memory editing of object states, relations, 3D boxes, and visual/context features.

  • Consolidation · Merging & deduplication

    Visual and spatial similarity re-identify recurring objects so observations update an existing entry instead of creating duplicates. For Embodied VideoAgent, this specifically supports merging or deduplication of overlapping entries of object states, relations, 3D boxes, and visual/context features.

Related systems