Multimodal Agent MemorySubmit a paper

BundleTrack

BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models

Bowen Wen, Kostas Bekris · 2021-08-01

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, depth, Video & streaming
Representation
Hybrid
Retained RGB-D keyframes and their geometric pose-graph state are explicitly linked and jointly consumed for correspondence construction and online pose refinement.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. memory pool of retained RGB-D keyframes selected from novel object views

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    BundleTrack retains historical RGB-D keyframes in a memory pool. For each new frame it selects up to 15 stored views with high mutual viewing overlap and supplies their raw feature and geometric correspondences to online pose-graph optimization.Full paper: Sec. IV Approach and Sec. IV-C Keyframe Selection · PDF pp. 3-4 · Figure 2

  2. SE(3) poses and correspondence structure attached to current and historical keyframes

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byMetadata / spatiotemporal retrieval, Structure traversal
    4. Source evidenceSource linked & recoverable
    The current frame and selected historical keyframes become pose-graph nodes linked by feature and dense geometric correspondence losses. Optimization returns the current pose and writes corrected historical poses back into the memory pool, where each state remains linked to its RGB-D keyframe.Full paper: Sec. IV-C Keyframe Selection and Sec. IV-D Online Pose Graph Optimization · PDF p. 4 · Equations 1-3

How memory changes (T2 operations)

  • Update · Rule-based update

    The first frame is retained, and a later frame is added to the memory pool when its pose is farther than a fixed angular threshold from every existing keyframe, enriching the stored views under a fixed novelty rule.

  • Update · Optimization-based update

    At each tracking step, iteratively reweighted Gauss-Newton optimization jointly refines the current and selected historical keyframe poses. The corrected historical poses are persisted in the memory pool and improve later tracking.

Related systems