Multimodal Agent MemorySubmit a paper

3D-Mem

3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning

Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, et al. · 2024-11-23

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, depth, text
Representation
Hybrid
Structured object/frontier state remains linked to retained snapshot images that the VLM directly inspects.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Memory Snapshots and Frontier Snapshots

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Selected raw multi-view RGB-D observations are retained as Memory Snapshots to preserve object and background detail.Full paper: 4.2. Episodic-Memory Embodied Q&A · PDF p. 7 · Figure 1
    Frontier Snapshots retain an image, navigable location, and the unexplored region it represents, and are directly supplied to the VLM.Full paper: 10.1. Implementation Details for A-EQA · PDF p. 11 · Figure 2

  2. object-to-snapshot assignments and frontier state

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Objects with categories and 3D locations are assigned to co-visible image snapshots; each snapshot links an object cluster to an observation.Full paper: 3.1.1. Memory Snapshot Formulation · PDF p. 4 · Algorithm 1
    A VLM ranks object categories for the instruction and filters snapshots that contain none of the selected categories.Full paper: 3.2.3. Memory Retrieval with Prefiltering · PDF p. 5 · Figure 3

How memory changes (T2 operations)

  • Update · Rule-based update

    At every exploration step, fixed procedures add detected objects, recluster affected snapshots, and add, modify, or remove frontier snapshots. For 3D-Mem, this specifically supports fixed-rule insertion or replacement of Memory Snapshots and Frontier Snapshots.

  • Consolidation · Merging & deduplication

    Snapshots that select the same frame candidate are merged after object clustering to maintain a compact set. For 3D-Mem, this specifically supports merging or deduplication of overlapping entries of Memory Snapshots and Frontier Snapshots.

Related systems