Multimodal Agent MemorySubmit a paper

MGA

MGA: Memory-Driven GUI Agent for Observation-Centric Interaction

Weihua Cheng, Ersheng Ni, Wenlong Wang, Yifei Sun, Junming Liu, Wangyu Shen, et al. · 2025-10-28

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. dynamic observation-centric structured memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Each decision uses the current screenshot, task-agnostic spatial information, and dynamically updated structured memory.Full paper: MGA: Memory-Driven GUI Agent for Observation-Centric Interaction · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During operation, MGA applies LLM-mediated memory editing to dynamic observation-centric structured memory; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Episodic / hierarchical summarization

    During operation, MGA applies episodic or hierarchical summarization to dynamic observation-centric structured memory; the memory state changes while the governing update policy remains fixed.

  • Revision · Rule-based state correction

    Independent context-rich state records limit propagation of stale trajectory errors. For MGA, this specifically supports fixed-rule correction of stored state of dynamic observation-centric structured memory.

Related systems