Multimodal Agent MemorySubmit a paper

EmbodiedRAG

EmbodiedRAG: Dynamic 3D Scene Graph Retrieval for Efficient and Scalable Robot Task Planning

Meghan Booker, Grayson Byrd, Bethany Kemp, Aurora Schmidt, Corban Rivera · 2024-10-31

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Time-varying 3D scene graph of entities, inter-entity edges and per-entity attributes, with each entity embedded as a retrievable document

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Vector-indexed collection
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    The robot maintains a 3DSG with entities O_t, edges E_t, entity attributes C^attr_{j,t} and edge attributes C^edge_{m,t} that semantically represents the observed state of the environment. Each entity is treated as a document whose page content is the object class / open-vocabulary label and whose metadata are its attributes; entity documents are embedded into a vector store (Chroma/FAISS). Retrieval is thresholded top-k similarity over entity abstractions, and the induced subgraph over the retrieved entities restores their relational structure, so the graph edges are traversed rather than merely stored. The self-query stage passes the planner's current thought tau_t instead of the raw task through an LLM to produce a structured query plus metadata filter, adding a planner-directed retrieval route. Serialized subgraphs handed to the planner contain only labels and filtered attributes; no RGB-D frame or image is linked back from a node, so the perceptual source is not recoverable from memory.Full paper (arXiv HTML v1): Sec. 4.3 3DSG as a retrievable document store; Sec. 4.5 self-query retrieval; Sec. 6 hardware experiments · HTML Sec. 4.3-4.5, Sec. 6 · Figure 1 (system pipeline); Table 2 (hardware trials)

How memory changes (T2 operations)

  • Update · Rule-based update

    As the robot is deployed in the environment, 3DSG construction begins and each newly perceived entity is embedded as a document and inserted into the retrievable store, so the memory grows and its content changes while the same memory is being queried for planning. The hardware experiment on the Spot quadruped with an RGB-D camera and ConceptGraphs shows an entity (the egg) that was recently populated into the full 3DSG being planned over. Actions change the environment, which in turn updates or adds documents. The insertion rule and the thresholded top-k retrieval procedure are fixed throughout; only memory content changes.

Related systems