Multimodal Agent MemorySubmit a paper

Multimodal Spatial Language Maps

Multimodal spatial language maps for robot navigation and manipulation

Chenguang Huang, Oier Mees, Andy Zeng, Wolfram Burgard · 2025-01-01

Read the paper

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. semantic map / spatial language map

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval, Structure traversal
    4. Source evidenceRetained but not accessible to the agent
    Autonomous exploration fuses foundation-model features with 3D reconstruction. LLMs translate commands into spatial goals localized in the map.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1 / method overview

  2. semantic map / spatial language map

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank
    3. Retrieved byMetadata / spatiotemporal retrieval, Structure traversal, Latent / parametric activation
    4. Source evidenceSource not retained
    Autonomous exploration fuses foundation-model features with 3D reconstruction. LLMs translate commands into spatial goals localized in the map.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1 / method overview

Related systems