Multimodal Agent MemorySubmit a paper

AutoDroid

AutoDroid: LLM-powered Task Automation in Android

Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, et al. · 2024-09-30

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
GUI & computer use
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. UI Transition Graph

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource not retained
    Offline exploration records UI states and actions in a directed transition graph used to synthesize app knowledge.Full paper: Memory · PDF p. 5 · Figure 5

  2. App Memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search
    4. Source evidenceSource not retained
    App Memory stores simulated-task entries linking textual functionality descriptions to UI-state and UI-element sequences.Full paper: OUR APPROACH: AUTODROID · PDF p. 4 · Figure 5
    The online agent retrieves the most similar simulated tasks and injects their navigation guidance into the prompt.Full paper: Exploration-based Memory Injection · PDF p. 6 · Algorithm 1

Related systems