Multimodal Agent MemorySubmit a paper

MementoGUI

MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

Ziyun Zeng, Hang Hua, Bocheng Zou, Mu Cai, Rogerio Feris, Jiebo Luo · 2026-05-18

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: Image, text, actions
Representation
Hybrid
Text summaries and ROI-level screenshots are jointly preserved as structured multimodal events.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. localized visual references for task-relevant interface events

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Working-memory events retain ROI-level visual evidence alongside concise summaries for later GUI decisions.Full-paper method / memory representation and retrieval · PDF p. 2

  2. event summaries and retrieved past trajectories

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    MementoCore performs step processing, compression, episodic writing, and learned episodic selection over reusable trajectories.Full paper: Model Output · PDF p. 4

How memory changes (T2 operations)

  • Update · Learned update module

    A trained controller selects online memory events and executes specialized working- and episodic-memory operators. For MementoGUI, this specifically supports a trained but frozen update module of localized visual references for task-relevant interface events.

  • Consolidation · Episodic / hierarchical summarization

    Selected interaction history is compressed into concise event summaries while preserving localized visual cues. For MementoGUI, this specifically supports episodic or hierarchical summarization of localized visual references for task-relevant interface events.

Related systems