Multimodal Agent MemorySubmit a paper

ConceptGraphs

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, et al. · 2023-09-28

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, depth, text
Representation
Hybrid
Geometric object state and the semantic-relational graph are linked in common object nodes and jointly support planning.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. 3D object map

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search, Structure traversal
    4. Source evidenceSource not retained
    Each object node contains a fused 3D point cloud and semantic feature vector, incrementally associated from posed RGB-D frames.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 2

  2. open-vocabulary 3D scene graph

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Object captions, locations, and inferred spatial relationships form nodes and edges used for language-grounded planning.Full paper: 6 We discard the (often noisy) tags produced by the image tagging model, · PDF p. 4
    The graph is exposed as structured object descriptions so a planner can select relevant objects and their 3D poses.Full paper: 6 We discard the (often noisy) tags produced by the image tagging model, · PDF p. 4

How memory changes (T2 operations)

  • Update · Rule-based update

    A fixed association-and-fusion pipeline incorporates each incoming RGB-D frame by updating matched objects or adding new ones. For ConceptGraphs, this specifically supports fixed-rule insertion or replacement of 3D object map.

Related systems