Multimodal Agent MemorySubmit a paper

RoboEXP

RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation

Hanxiao Jiang, Binghao Huang, Ruihai Wu, Zhuoran Li, Shubham Garg, Hooshang Nayyeri, et al. · 2024-02-23

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. metric and semantic scene state

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource linked & recoverable
    The graph retains geometry and semantics.Full-paper method / memory representation and retrieval · PDF p. 3 · Figure 1

  2. action-conditioned entity relations

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Interactive exploration adds action-conditioned relationships.Full-paper method / memory representation and retrieval · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Update · Rule-based update

    New entities and relations are incrementally added. For RoboEXP, this specifically supports fixed-rule insertion or replacement of metric and semantic scene state.

  • Revision · Rule-based state correction

    Interaction corrects object state records. For RoboEXP, this specifically supports fixed-rule correction of stored state of metric and semantic scene state.

  • Reorganization · Relinking & reindexing

    During operation, RoboEXP applies relinking or reindexing of stored items to metric and semantic scene state; the memory state changes while the governing update policy remains fixed.

  • Forgetting · Discrete eviction

    During operation, RoboEXP applies explicit deletion or eviction of stored items to metric and semantic scene state; the memory state changes while the governing update policy remains fixed.

Related systems