Multimodal Agent MemorySubmit a paper

KnowAct-GUIClaw

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

Yunxin Li, Jinchao Li, Baotian Hu, Min Zhang, Shibo Su, Zhenran Xu, et al. · 2026-07-15

Read the paperPDF

Management
T3 Self-Evolving Memory
Feedback during deployment changes the memory-management policy itself.
Primary modality
GUI & computer use
Also: Image, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. trajectory-derived experience memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Success and failure traces become concise textual policies that are recalled by semantic similarity and supplied to later routing, decomposition, layout reasoning, and recovery.Full paper: §§4.2 and 4.5 · HTML full text · Figure 3

  2. state-validated GUI skill and shortcut store

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Each skill stores app and platform scope, parameters, ordered GUI steps, reliability fields, and visual or deterministic state contracts; app-scoped candidates are retrieved by lexical and embedding signals and checked for applicability.Full paper: §§4.4–4.5 · HTML full text · Figure 2

How memory changes (T2 operations)

  • Consolidation · Experience abstraction

    Post-run reflection turns GUI trajectories, screenshots, action parameters, and inferred state contracts into concise lessons and reusable declarative procedures.

  • Update · Prompted LLM/VLM record update

    A vision-language extraction and evolution process writes accepted skills and edits an identified failed skill in place while enforcing action, parameter, scope, and state-validation constraints.

  • Consolidation · Merging & deduplication

    The experience-memory inducer limits retained items per task and drops near-duplicate lessons within the same application.

How the policy evolves (T3)

  • Retrieval policy

    When a reused skill fails during GUI execution, reflection uses the failed step, screen evidence, error, original skill, and prior feedback to update that skill in place, including narrowing its retrieval description and refreshing targets or state contracts; the persistent edited skill is retrieved and validated on later subtasks.

Related systems