Multimodal Agent MemorySubmit a paper

MoMa-LLM

Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation

Daniel Honerkamp, Martin Büchner, Fabien Despinoy, Tim Welschehold, Abhinav Valada · 2024-07-31

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. open-vocabulary 3D map state

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceRetained but not accessible to the agent
    RGB-D detections update a 3D map as the mobile manipulator explores.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1

  2. dynamic hierarchical scene graph

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Rooms, objects, and semantic-spatial relations are dynamically updated and supplied to the LLM planner.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Update · Rule-based update

    The scene graph is updated after new perceptual observations. For MoMa-LLM, this specifically supports fixed-rule insertion or replacement of open-vocabulary 3D map state.

  • Reorganization · Relinking & reindexing

    Object-room and spatial relations change as exploration discovers new evidence. For MoMa-LLM, this specifically supports relinking or reindexing of stored items of open-vocabulary 3D map state.

Related systems