Multimodal Agent MemorySubmit a paper

Vision to Geometry / 3DSPMR

Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration

Zhongyi Cai, Yi Du, Chen Wang, Yu Kong · 2026-03-18

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Compact representative spatial memory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval, Latent / parametric activation
    4. Source evidenceSource not retained
    Stores representative spatial data rather than all redundant observations.; Uses FoV-based constraints as explicit geometric cues for memory construction.; Memory is accumulated from previous explorations and carried into later tasks.. spatial memory reused across sequential tasks to guide exploration.Full paper: Our Method · PDF p. 14 · Figure 1 / method overview

  2. FoV coverage information

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank, Vector-indexed collection
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval, Latent / parametric activation
    4. Source evidenceSource not retained
    Stores representative spatial data rather than all redundant observations.; Uses FoV-based constraints as explicit geometric cues for memory construction.; Memory is accumulated from previous explorations and carried into later tasks.. spatial memory reused across sequential tasks to guide exploration.Full paper: Our Method · PDF p. 14 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, Vision to Geometry / 3DSPMR applies fixed-rule insertion or replacement to Compact representative spatial memory; the memory state changes while the governing update policy remains fixed.

  • Reorganization · Relinking & reindexing

    During operation, Vision to Geometry / 3DSPMR applies relinking or reindexing of stored items to Compact representative spatial memory; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Merging & deduplication

    During operation, Vision to Geometry / 3DSPMR applies merging or deduplication of overlapping entries to Compact representative spatial memory; the memory state changes while the governing update policy remains fixed.

Related systems