Multimodal Agent MemorySubmit a paper

RGB-Only Reconstruction and MPC

RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

Zhenggang Tang, Balakumar Sundaralingam, Jonathan Tremblay, Bowen Wen, Ye Yuan, Stephen Tyree, et al. · 2022-10-21

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Embodied & 3D
Also: Image
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. fixed NeRF-derived scene mesh and Euclidean full signed distance field used by the manipulator controller

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource not retained
    Multiple RGB observations are consolidated into a per-scene NeRF; a polygonal mesh and bounding-volume hierarchy are extracted, and signed distances are queried to form an ESDF suitable for downstream robotic planning.Full paper: Sec. III-A Neural Scene Reconstruction from RGB Images · PDF p. 3 · Figure 1; Eq. 1-3
    After reconstruction, the scene mesh is loaded into a sampling-based model-predictive controller. The controller repeatedly queries the fixed signed-distance representation to generate collision-free robot motions toward later target poses.Full paper: Sec. III-B Manipulator Control Problem · PDF p. 3 · Figure 1

Related systems