Multimodal Agent MemorySubmit a paper

MobileGPT

MobileGPT: Augmenting LLM with Human-like App Memory for Mobile Task Automation

Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y. Ko, et al. · 2024-10-16

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: Image, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. screens, reusable subtasks, and action procedures

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store, Graph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    App memory organizes screens and derived action procedures into reusable, parameterizable subtasks that can be selected and recomposed for new goals.Full paper: System Design · PDF p. 3

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    The LLM selects useful explored paths and derives a task procedure that is stored in app memory. For MobileGPT, this specifically supports LLM-mediated memory editing of screens, reusable subtasks, and action procedures.

  • Consolidation · Experience abstraction

    Explored action sequences are decomposed into modular subtask procedures for reuse across tasks. For MobileGPT, this specifically supports abstraction of episodes into reusable experience of screens, reusable subtasks, and action procedures.

  • Revision · Rule-based state correction

    During operation, MobileGPT applies fixed-rule correction of stored state to screens, reusable subtasks, and action procedures; the memory state changes while the governing update policy remains fixed.

Related systems