Multimodal Agent MemorySubmit a paper

UI-TARS

UI-TARS: Pioneering Automated GUI Interaction with Native Agents

Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, et al. · 2025-01-21

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
GUI & computer use
Also: Image, text
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. short-term memory: retained history of prior thoughts and actions plus a recency window of GUI screenshots

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    At each step the model predicts P(t_n, a_n | instruction, t_1, a_1, ..., (o_{n-i}, t_{n-i}, a_{n-i})_{i=1..N}, o_n). Section 4.1 states that the full history of previous actions and thoughts is retained as short-term memory, while raw screenshot observations are truncated to the last N to fit the 32k sequence budget. The retained item is therefore an accumulating sequential record of self-generated System-2 thoughts and executed actions, with a recency-selected window of the underlying screenshots still directly present.Full paper: Section 4.1 (inference formulation)
    Section 3.1 defines short-term memory as a temporary repository for task-specific information capturing the agent's immediate context, i.e. the paper explicitly designates this history as a named memory component of the architecture rather than an incidental prompt buffer.Full paper: Section 3.1 (agent memory taxonomy)

  2. long-term operational experience internalized in the native agent model's weights

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inUnclear
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Section 3.1 contrasts framework agents, which keep long-term memory in external stores, with native agent models that encode long-term operational experience of tasks within their internal parameters, converting the observable interaction process into implicit parameterized storage. This parametric internalization of GUI interaction experience is the paper's central design claim and is what the model consults when acting.Full paper: Section 3.1 (native agent model vs framework agent)
    Experience enters the parameters through an iterative loop: model M_n executes tasks on hundreds of virtual machines, traces are filtered by rule-based heuristics, VLM scoring and human review, annotators mark the erroneous step and label the corrected thought and action, and the model is retrained by reflection tuning. The loop is run for multiple rounds to produce successive model versions.Full paper: Section 4.5 (online trace bootstrapping and reflection tuning)

How memory changes (T2 operations)

  • Update · Rule-based update

    During task execution the short-term memory is extended at every step by a fixed append rule: the newly generated thought and executed action are written into the retained history and the screenshot window is advanced to the last N observations. Content and window membership change during inference while the governing update rule itself is fixed.

Related systems