Multimodal Agent MemorySubmit a paper

MAGNET

MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution

Libo Sun, Jiwen Zhang, Siyuan Wang, Zhongyu Wei · 2026-01-27

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. stationary visual-semantic memory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceRetained but not accessible to the agent
    Stationary memory links diverse visual features to stable functional semantics.Full paper: MAGNET Framework · PDF p. 2 · Figure 1

  2. procedural intent memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Procedural memory captures stable task intents across changing GUI workflows.Full paper: MAGNET Framework · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, MAGNET applies fixed-rule insertion or replacement to stationary visual-semantic memory; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Merging & deduplication

    During operation, MAGNET applies merging or deduplication of overlapping entries to stationary visual-semantic memory; the memory state changes while the governing update policy remains fixed.

  • Reorganization · Relinking & reindexing

    Visual-semantic and procedural associations are updated across interface changes. For MAGNET, this specifically supports relinking or reindexing of stored items of stationary visual-semantic memory.

  • Revision · Evidence-weighted belief revision

    Frequently accessed knowledge is prioritized while both memories are refined. For MAGNET, this specifically supports evidence-weighted revision of stored belief or confidence of stationary visual-semantic memory.

Related systems