Multimodal Agent MemorySubmit a paper

Embodied-RAG

Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation

Quanting Xie, So Yeon Min, Pengliang Ji, Yue Yang, Tianyi Zhang, Kedi Xu, et al. · 2024-09-26

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. hierarchical semantic forest

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search, Structure traversal
    4. Source evidenceSource linked & recoverable
    Language descriptions are organized at multiple spatial and semantic resolutions in a searchable semantic forest.Full paper: A. Retrieval and Generation · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Episodic / hierarchical summarization

    Perceptual experience is consolidated into descriptions at multiple levels of detail. For Embodied-RAG, this specifically supports episodic or hierarchical summarization of hierarchical semantic forest.

  • Reorganization · Relinking & reindexing

    Descriptions are inserted into a spatial-semantic hierarchy for context-sensitive recall. For Embodied-RAG, this specifically supports relinking or reindexing of stored items of hierarchical semantic forest.

Related systems