Multimodal Agent MemorySubmit a paper

SE-GA

SE-GA: Memory-Augmented Self-Evolution for GUI Agents

Shilong Jin, Lanjun Wang, Zhuosheng Zhang · 2026-05-16

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: Image, text
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. episodic, semantic, and experiential GUI memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    TTME retrieves hierarchical episodic, semantic, and experiential memory at inference.Full paper: 2.2. Memory Mechanisms · PDF p. 2

  2. LoRA-adapted GUI policy conditioned on accumulated interaction experience

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    MASE uses accumulated GUI interaction experience to adapt the agent policy with LoRA, internalizing reusable experience in parameters.Full paper: Model · PDF p. 8

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, SE-GA applies fixed-rule insertion or replacement to episodic, semantic, and experiential GUI memory; the memory state changes while the governing update policy remains fixed.

  • Forgetting · Discrete eviction

    During operation, SE-GA applies explicit deletion or eviction of stored items to episodic, semantic, and experiential GUI memory; the memory state changes while the governing update policy remains fixed.

Related systems