Multimodal Agent MemorySubmit a paper

Planning from Imagination

Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigation

Yiyuan Pan, Yunzhe Xu, Zhe Liu, Hesheng Wang · 2025-01-01

Read the paper

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. episodic / trajectory memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource not retained
    Mixed-granularity memory updated during navigation and expanded through episodic simulation. memory/imagined viewpoints guide waypoint and action selection.Full paper: Method · PDF p. 2 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, Planning from Imagination applies fixed-rule insertion or replacement to episodic / trajectory memory; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Merging & deduplication

    During operation, Planning from Imagination applies merging or deduplication of overlapping entries to episodic / trajectory memory; the memory state changes while the governing update policy remains fixed.

  • Reorganization · Relinking & reindexing

    Topological memory updates at each navigation step. For Planning from Imagination, this specifically supports relinking or reindexing of stored items of episodic / trajectory memory.

Related systems