Multimodal Agent MemorySubmit a paper

AutoMMemo

Learning to Learn from Multimodal Experience

Xingyu Sui, Weixiang Zhao, Yongxin Tang, Yanyan Zhao, Yang Wu, Dandan Tu, et al. · 2026-05-16

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: Image, text
Representation
Hybrid
source-linked executable multimodal memory program

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. multimodal episode recorders

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource directly available
    Episode recorders retain visual observations, actions, messages, metadata, and outcomes.Full paper: Additional Algorithm Details · PDF p. 13

  2. executable update/retrieve memory program

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Hierarchical multi-tier store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    An executable Python memo program defines how episodes are stored, organized, and retrieved.Full paper: Additional Algorithm Details · PDF p. 13

How memory changes (T2 operations)

  • Reorganization · Relinking & reindexing

    Candidate programs can change indexing, state-action associations, ranking, and payload structure. For AutoMMemo, this specifically supports relinking or reindexing of stored items of multimodal episode recorders.

  • Consolidation · Merging & deduplication

    Program mutation can remove noisy and redundant memory content. For AutoMMemo, this specifically supports merging or deduplication of overlapping entries of multimodal episode recorders.

Related systems