Multimodal Agent MemorySubmit a paper

MEM

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, et al. · 2026-03-04

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. short-horizon video memory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceRetained but not accessible to the agent
    A video encoder compresses dense recent observations for fine-grained control.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1

  2. long-horizon language memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource not retained
    A high-level policy maintains compressed language state for task progress over up to fifteen minutes.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Update · Learned update module

    The high-level policy autoregressively updates language memory during control. For MEM, this specifically supports a trained but frozen update module of short-horizon video memory.

  • Consolidation · Episodic / hierarchical summarization

    Long-horizon events are compressed into semantic task-state summaries. For MEM, this specifically supports episodic or hierarchical summarization of short-horizon video memory.

Related systems