Multimodal Agent MemorySubmit a paper

Voyager

Voyager: An Open-Ended Embodied Agent with Large Language Models

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, et al. · 2023-05-25

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. executable skill library

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Temporally extended executable code skills are stored, retrieved, composed, and reused across Minecraft worlds.Full paper: Update · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    Environment feedback, execution errors, and self-verification revise skill programs. For Voyager, this specifically supports LLM-mediated memory editing of executable skill library.

  • Consolidation · Experience abstraction

    Verified programs are admitted as reusable behavior skills. For Voyager, this specifically supports abstraction of episodes into reusable experience of executable skill library.

Related systems