Multimodal Agent MemorySubmit a paper

UI-Mem

UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents

Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, et al. · 2026-02-05

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
GUI & computer use
Also: Image, text, actions
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. workflows, subtask skills, and failure-pattern templates

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Raw trajectories are abstracted into parameterized high-level workflows, executable subtask skills, and failure patterns retrieved and instantiated for new tasks.Full paper: 4.1. Implementation Details · PDF p. 6 · Figure 2

Related systems