Multimodal Agent MemorySubmit a paper

Optimus-1

Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, Liqiang Nie · 2024-08-07

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. abstracted multimodal experience pool

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource directly available
    Historical multimodal episodes are retained as references for in-context reflection.Full paper: Implementation Details · PDF p. 20 · Figure 1

  2. hierarchical directed knowledge graph

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    World knowledge is transformed into a hierarchical directed graph for planning.Full-paper method / memory representation and retrieval · PDF p. 4 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Experience abstraction

    Historical trajectories are summarized into reusable multimodal experiences. For Optimus-1, this specifically supports abstraction of episodes into reusable experience of abstracted multimodal experience pool.

  • Consolidation · Episodic / hierarchical summarization

    World knowledge and episodes are organized at multiple levels. For Optimus-1, this specifically supports episodic or hierarchical summarization of abstracted multimodal experience pool.

  • Forgetting · Discrete eviction

    During operation, Optimus-1 applies explicit deletion or eviction of stored items to abstracted multimodal experience pool; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Merging & deduplication

    During operation, Optimus-1 applies merging or deduplication of overlapping entries to abstracted multimodal experience pool; the memory state changes while the governing update policy remains fixed.

Related systems