Multimodal Agent MemorySubmit a paper

DovSG

Dynamic Open-Vocabulary 3D Scene Graphs for Long-term Language-Guided Mobile Manipulation

Zhijie Yan, Shufei Li, Zuoxu Wang, Lixiu Wu, Han Wang, Jun Zhu, et al. · 2025-03-19

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, depth, text
Representation
Hybrid
Both operative paths remain within the source-preserving/modality-native parent family.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. 3D object nodes with geometric and semantic features

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceRetained but not accessible to the agent
    Each object node carries fused geometry and open-vocabulary semantic features, with spatial relations represented by graph edges.Full-paper method / memory representation and retrieval · PDF p. 2

  2. historical RGB-D images and poses used for relocalization

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource directly available
    The system retrieves similar historical 2D images and poses, matches them to the current view, and refines robot pose before local graph updates.Full-paper method / memory representation and retrieval · PDF p. 2

  3. dynamic 3D object-relation scene graph

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Object nodes and spatial relations form a dynamic 3D scene graph whose updated structure is traversed for planning and remains linked to geometric and historical visual evidence.Full-paper method / memory representation and retrieval · PDF p. 2

How memory changes (T2 operations)

  • Update · Rule-based update

    After interaction, fixed geometric checks locally update object features and the affected scene graph rather than reconstructing the full scene. For DovSG, this specifically supports fixed-rule insertion or replacement of 3D object nodes with geometric and semantic features.

  • Reorganization · Relinking & reindexing

    Obsolete indices are removed and low- and high-level graph relations are rebuilt for changed regions. For DovSG, this specifically supports relinking or reindexing of stored items of 3D object nodes with geometric and semantic features.

  • Revision · Rule-based state correction

    New RGB-D evidence is checked against fixed depth and color criteria; obsolete points are removed and affected graph state is recomputed. For DovSG, this specifically supports fixed-rule correction of stored state of 3D object nodes with geometric and semantic features.

Related systems