Multimodal Agent MemorySubmit a paper

Agent S

Agent S: An Open Agentic Framework that Uses Computers Like a Human

Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang · 2024-10-10

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. episodic computer-use memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource linked & recoverable
    Specific GUI action sequences are summarized and retained as episodic experience.Full paper: Memory · PDF p. 4 · Figure 1

  2. narrative task knowledge

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Patterns across experiences are distilled into higher-level narrative knowledge for hierarchical planning.Full paper: System · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Experience abstraction

    Successful experiences are distilled into reusable narrative knowledge. For Agent S, this specifically supports abstraction of episodes into reusable experience of episodic computer-use memory.

  • Consolidation · Episodic / hierarchical summarization

    Task executions are summarized before storage. For Agent S, this specifically supports episodic or hierarchical summarization of episodic computer-use memory.

Related systems