Multimodal Agent MemorySubmit a paper

ActionEngine

ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory

Hongbin Zhong, Fazle Faisal, Luis França, Tanakorn Leesatapornwongsa, Adriana Szekeres, Kexin Rong, et al. · 2026-02-24

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. updatable GUI state-machine memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Crawler-validated GUI states and action templates form a state machine used to synthesize executable programs.Full paper: System Overview · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Update · Rule-based update

    Execution failures trigger fixed vision-based re-grounding and persistent memory updates. For ActionEngine, this specifically supports fixed-rule insertion or replacement of updatable GUI state-machine memory.

  • Revision · Rule-based state correction

    Failed actions are repaired and their state-machine entries corrected. For ActionEngine, this specifically supports fixed-rule correction of stored state of updatable GUI state-machine memory.

Related systems