Multimodal Agent MemorySubmit a paper

HyMEM

Hybrid Self-evolving Structured Memory for GUI Agents

Sibo Zhu, Wenyi Wu, Kun Zhou, Stephen Wang, Biwei Huang · 2026-03-11

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. symbolic GUI memory graph

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource linked & recoverable
    Discrete high-level symbolic nodes support multi-hop retrieval and working-memory refresh.Full paper: Memory Size · PDF p. 8 · Figure 1

  2. continuous trajectory embeddings

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Latent / parametric activation
    4. Source evidenceSource not retained
    Continuous trajectory embeddings are linked to symbolic nodes and supplied to the GUI agent.Full paper: Hybrid Structured Memory · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Merging & deduplication

    Fixed node operations merge or replace overlapping GUI experiences. For HyMEM, this specifically supports merging or deduplication of overlapping entries of symbolic GUI memory graph.

  • Reorganization · Relinking & reindexing

    Node updates reorganize the symbolic graph and its trajectory links. For HyMEM, this specifically supports relinking or reindexing of stored items of symbolic GUI memory graph.

  • Revision · Rule-based state correction

    On-the-fly working-memory refresh updates the active task state. For HyMEM, this specifically supports fixed-rule correction of stored state of symbolic GUI memory graph.

Related systems