Multimodal Agent MemorySubmit a paper

DynaMem

DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation

Peiqi Liu, Zhanqiu Guo, Mohit Warke, Soumith Chintala, Chris Paxton, Nur Muhammad Mahi Shafiullah, et al. · 2025-05-29

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, depth, text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. dynamic spatio-semantic voxel point cloud

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Each voxel retains 3D location, an observation count, a count-weighted semantic feature, latest time, and latest contributing image identifier.Full paper: Update rule · PDF p. 4 · Figure 2
    Text-to-feature similarity selects a voxel and its latest associated image for object verification.Full paper: Update rule · PDF p. 4 · Figure 4

  2. source images referenced by voxels

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval, Similarity search
    4. Source evidenceSource linked & recoverable
    The latest image identifier stored with a selected voxel resolves back to an RGB observation used by the multimodal verifier.Full paper: Update rule · PDF p. 4 · Figure 4

How memory changes (T2 operations)

  • Update · Rule-based update

    A fixed voxel-assignment rule adds new points and updates feature, count, time, and source-image metadata. For DynaMem, this specifically supports fixed-rule insertion or replacement of dynamic spatio-semantic voxel point cloud.

  • Revision · Rule-based state correction

    Ray casting removes voxels that should be visible in the current camera frustum but are absent from the new depth observation. For DynaMem, this specifically supports fixed-rule correction of stored state of dynamic spatio-semantic voxel point cloud.

Related systems