Multimodal Agent MemorySubmit a paper

AesopAgent

AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production

Jiuniu Wang, Zehua Du, Yuyuan Zhao, Bo Yuan, Kexiang Wang, Jian Liang, et al. · 2024-03-12

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text, Image, audio
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. expert experience and professional knowledge

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource not retained
    The RAG layer accumulates expert experience and professional knowledge for future production runs.Full-paper method / memory representation and retrieval · PDF p. 3 · Figure 1

  2. optimized prompts and utility settings

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Accumulated experience is used to iteratively optimize prompts, utilities, and the video-generation workflow.Full-paper method / memory representation and retrieval · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During operation, AesopAgent applies LLM-mediated memory editing to expert experience and professional knowledge; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Experience abstraction

    Production experience is accumulated as reusable expert knowledge. For AesopAgent, this specifically supports abstraction of episodes into reusable experience of expert experience and professional knowledge.

Related systems