Multimodal Agent MemorySubmit a paper

MobiMem

Beyond Training: Enabling Self-Evolution of Agents with MobiMem

Zibin Liu, Cheng Zhang, Xi Zhao, Yunfei Feng, Bingyu Bai, Dahu Feng, et al. · 2025-12-15

Read the paperPDF

Management
T3 Self-Evolving Memory
Feedback during deployment changes the memory-management policy itself.
Primary modality
GUI & computer use
Also: Image, Audio & speech, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. concept-entity user profile memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search, Structure traversal
    4. Source evidenceSource not retained
    Profile Memory organizes user facts, preferences, and behavioral patterns in a concept-entity distance graph for embedding retrieval and graph traversal.Full paper: §§3 and 4.1 · HTML full text · Figure 2

  2. multi-level execution-template memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Experience Memory decomposes completed workflows into reusable templates with invariant control logic and parameter slots that a Task Rewriter retrieves and instantiates for new tasks.Full paper: §§3 and 4.2 · HTML full text · Figure 2

  3. ActTree and ActChain replay memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    ActTree and ActChain retain UI states, actions, and template-to-action mappings for verified prefix or prefix-suffix replay, with links back to recorded GUI execution state.Full paper: §§4.3 and 5.2 · HTML full text · Figure 5

How memory changes (T2 operations)

  • Consolidation · Experience abstraction

    The Experience Generator summarizes deployed task traces into reusable templates and maps template steps to recorded action sequences.

  • Update · Prompted LLM/VLM record update

    After a user correction, the Experience Generator compares the original plan with the corrected execution and distills the cause into an improved persistent template.

  • Consolidation · Merging & deduplication

    AgentRR merges a newly recorded GUI path into matching ActTree states and action edges and updates the associated task list rather than duplicating the path.

  • Revision · Rule-based state correction

    Before replay, cached actions are checked against the current UI hierarchy; a mismatch rolls back the stale action, falls back to replanning, and records an alternative path or refreshed action memory.

How the policy evolves (T3)

  • Retrieval policy

    During deployment, explicit user interruptions and manual GUI corrections become feedback: the Experience Generator diagnoses the deviation and persistently evolves the reusable execution template, which the Task Rewriter and action-replay mechanisms select for future tasks to avoid the same error.

Related systems