Multimodal Agent MemorySubmit a paper

VLM-MSGraph

VLM-MSGraph: Vision Language Model-enabled Multi-hierarchical Scene Graph for robotic assembly

S. Li, Z. Yan, Z. Wang, Y. Gao · 2025-01-01

Read the paper

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Embodied & 3D
Also: text, Image
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. assembly sequence triplets

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store, Hierarchical multi-tier store
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    High-level triplets generated by VLM agents and low-level 3D object instances stored as manipulation memory. VLM agents and retrieval methods learn/use assembly sequences; low-level 3D memory supports manipulation execution.Publisher abstract (full text unavailable) · Publisher abstract; full text unavailable · Figure 1 / method overview

  2. 3D object instances

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Hierarchical multi-tier store
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    High-level triplets generated by VLM agents and low-level 3D object instances stored as manipulation memory. VLM agents and retrieval methods learn/use assembly sequences; low-level 3D memory supports manipulation execution.Publisher abstract (full text unavailable) · Publisher abstract; full text unavailable · Figure 1 / method overview

Related systems