Multimodal Agent MemorySubmit a paper

PAL-UI

PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents

Zikang Liu, Junyi Li, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-rong Wen · 2025-10-04

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: Image, text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. historical screenshots

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource directly available
    A look-back tool recalls particular past screenshots.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1

  2. observation cues and action outcomes

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    A dual-level summarizer records observation cues and action outcomes linked to the trajectory.Full paper: History Memory · PDF p. 4 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Episodic / hierarchical summarization

    A fixed summarizer compresses the growing GUI trajectory. For PAL-UI, this specifically supports episodic or hierarchical summarization of historical screenshots.

  • Update · Rule-based update

    During operation, PAL-UI applies fixed-rule insertion or replacement to historical screenshots; the memory state changes while the governing update policy remains fixed.

Related systems