Multimodal Agent MemorySubmit a paper

Open Scene Graphs

Open Scene Graphs for Open World Object-Goal Navigation

Joel Loo, Zhanxin Wu, David Hsu · 2024-07-02

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. topo-semantic open-set scene information

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Open Scene Graphs retain and organize open-set scene information during object-goal navigation. LLM reasoning and pathfinding query/use the OSG for object-goal navigation; explicit retrieval is partial.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During operation, Open Scene Graphs applies LLM-mediated memory editing to topo-semantic open-set scene information; the memory state changes while the governing update policy remains fixed.

  • Reorganization · Relinking & reindexing

    During operation, Open Scene Graphs applies relinking or reindexing of stored items to topo-semantic open-set scene information; the memory state changes while the governing update policy remains fixed.

Related systems