Multimodal Agent MemorySubmit a paper

Mobile-Agent-E

Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks

Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, et al. · 2025-01-28

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: Image, text
Representation
Hybrid
Verbalized Tips and structured executable Shortcuts are distinct operative paths that are jointly supplied to the Manager and Operator in the same decision prompt.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Tips: natural-language lessons distilled from previous trial-and-error on the phone GUI

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Tips are 'general guidance on effective interactions and lessons learned from previous trial-and-error experiences', held in natural language in the persistent long-term memory and injected into the Operator's prompt; at scale an Experience Retriever agent 'select[s] only the relevant Tips and Shortcuts for the current task'. Only the verbalized lesson is kept, not the underlying screenshots.Full paper (HTML v2): 2.2 Self-Evolution Module; 4.4 Scaling to larger task pools; Appendix G · HTML full text · Figure 1 (framework overview); Figure 13 (generated Tips)

  2. Shortcuts: reusable JSON functions composed of sequences of atomic GUI operations with preconditions

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Shortcuts are 'reusable, executable functions composed of sequences of atomic operations tailored for recurring subroutines', stored in a predefined JSON format with an explicit precondition because 'a Shortcut may only be applicable in certain states'. They encode ordered GUI action structure (Tap(x,y), Type(text), ...) over app states and are supplied to the Manager for high-level planning and to the Operator for execution.Full paper (HTML v2): 2.2 Self-Evolution Module; 2.1 (Manager/Operator inputs, Eq. 1); Appendix F · HTML full text · Figure 12 (full list of generated Shortcuts)

  3. Working memory: Progress Status, Notes, action history and error history maintained across steps of a task

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource not retained
    The Perceptor turns each screenshot into 'a fine-grained list of texts and icons, along with their coordinates'; the Notetaker aggregates 'task-relevant information' such as prices or phone numbers into Notes, and the Action Reflector updates the Progress Status after each action. These verbalized records are explicitly designed working memory that is re-read by the Manager and Operator at later steps of the same task, and they are also the input the Experience Reflectors consume at task end. They reset between tasks.Full paper (HTML v2): 2.1 Framework (Perceptor, Action Reflector, Notetaker; Eq. 7) · HTML full text · Figure 1

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    At the end of each deployed task the two Experience Reflectors take the existing long-term memory plus the task's action and error history as input and emit 'newly generated Shortcuts in a predefined JSON format and updated Tips in natural language'. In the '+Evo' setting the agent runs sequentially over the task pool and the memory carried into task k+1 is the memory rewritten after task k. The reflector prompts (the governing update policy) are fixed.

  • Consolidation · Experience abstraction

    Concrete deployed GUI trajectories are abstracted into two reusable forms: generalized natural-language Tips and parameterized executable Shortcuts over atomic operations, which are then reused on later tasks.

  • Update · Rule-based update

    Within a task, Progress Status and Notes are appended/overwritten after each verified action under a fixed protocol, and the resulting state is consumed by the next planning and action step.

Related systems