Multimodal Agent MemorySubmit a paper

Scene Graph Memory

Modeling Dynamic Environments with Scene Graph Memory

Andrey Kurenkov, Michael Lingelbach, Tanmay Agarwal, Emily Jin, Chengshu Li, Ruohan Zhang, et al. · 2023-06-12

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. agent accumulated observations of rooms, objects, and temporal relations in dynamic environments

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval, Structure traversal
    4. Source evidenceSource not retained
    Scene Graph Memory aggregates observed nodes and edges with time references; NEP predicts likely object locations. NEP extracts information from SGM to predict object locations and guide efficient search; explicit retrieval is model-mediated.Full paper: 4. Method · PDF p. 3 · Figure 1 / method overview

How memory changes (T2 operations)

  • Revision · Rule-based state correction

    During operation, Scene Graph Memory applies fixed-rule correction of stored state to agent accumulated observations of rooms, objects, and temporal relations in dynamic environments; the memory state changes while the governing update policy remains fixed.

  • Reorganization · Relinking & reindexing

    SGM is updated as the agent observes subsets of the changing scene graph over time. For Scene Graph Memory, this specifically supports relinking or reindexing of stored items of agent accumulated observations of rooms, objects, and temporal relations in dynamic environments.

Related systems