Multimodal Agent MemorySubmit a paper

BundleSDF

BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, et al. · 2023-03-24

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, depth, Video & streaming
Representation
Hybrid
Retained RGB-D frames, their pose-graph state, and the neural object field form a closed operative loop: frames train the field, the field refines stored poses, and both support subsequent tracking and reconstruction.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. memory pool of retained posed RGB-D frames selected for multi-view diversity

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    BundleSDF retains informative historical RGB-D observations with their poses in a memory pool. A subset selected for viewing overlap supports each online pose graph, while all stored frames are later consumed by the concurrently learned object field.Full paper: Sec. 3.2 Memory Pool and Sec. 3.3 Online Pose Graph Optimization · PDF pp. 3-4 · Figure 2

  2. coupled keyframe pose graph and per-video neural signed-distance field encoding object geometry and appearance

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Latent / parametric activation
    4. Source evidenceSource linked & recoverable
    A pose graph jointly refines current and historical keyframe poses, while a per-video neural object field learns geometry and appearance from all memory frames. Optimized poses are written back to the pool, and the learned SDF supplies a unary loss to subsequent pose-graph updates; both states remain linked to the retained source frames.Full paper: Sec. 3.3 Online Pose Graph Optimization and Sec. 3.4 Neural Object Field · PDF pp. 4-5 · Figure 2; Equations 1 and 4-7

How memory changes (T2 operations)

  • Update · Rule-based update

    The first observation initializes the memory pool, and later posed frames are admitted only when their viewpoint adds sufficient multi-view diversity under a fixed threshold rule.

  • Update · Optimization-based update

    During online tracking, Gauss-Newton pose-graph optimization corrects stored keyframe poses and concurrent gradient training updates the neural object field. The corrected poses and learned SDF are reused by later tracking steps, and training repeats as new memory frames arrive.

Related systems