Multimodal Agent MemorySubmit a paper

SOMA

Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

Pengteng Li, Weiyu Guo, He Zhang, Tiefu Cai, Xiao He, Yandong Guo, et al. · 2026-05-21

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. multi-view spatial-semantic map

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource linked & recoverable
    Scans are aggregated into persistent spatial memory.Full paper: 3. Method · PDF p. 3 · Figure 1

  2. instruction-relevant spatial cues

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Retrieval activates object and layout cues.Full paper: Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Revision · Rule-based state correction

    Dynamic refinement maintains global consistency. For SOMA, this specifically supports fixed-rule correction of stored state of multi-view spatial-semantic map.

Related systems