Multimodal Agent MemorySubmit a paper

Latent Spatial Memory

Latent Spatial Memory for Video World Models

Weijie Wang, Haoyu Zhao, Yifan Yang, Feng Chen, Zeyu Zhang, Yefei He, et al. · 2026-06-08

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text, Image, Embodied & 3D
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. persistent 3D scene information for spatially consistent video generation

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval, Latent / parametric activation
    4. Source evidenceSource not retained
    Diffusion latent tokens tied to world-space coordinates in a persistent 3D cache. queries the persistent cache by projecting latent-attributed 3D points onto target camera grids with z-buffering and ControlNet-style injection.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1 / method overview

  2. persistent 3D scene information for spatially consistent video generation

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inVector-indexed collection
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval, Latent / parametric activation
    4. Source evidenceSource not retained
    Diffusion latent tokens tied to world-space coordinates in a persistent 3D cache. queries the persistent cache by projecting latent-attributed 3D points onto target camera grids with z-buffering and ControlNet-style injection.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Rule-based update

    Initialize-readout-update loop persists static scene information across chunks; generated frames can update the latent cache; no self-improving policy. For Latent Spatial Memory, this specifically supports fixed-rule insertion or replacement of persistent 3D scene information for spatially consistent video generation.

Related systems