Multimodal Agent MemorySubmit a paper

SkeMex

Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory

Haoran Sun, Wenjie Li, Yujie Zhang, Zekai Lin, Fanrui Zhang, Kaitao Chen, et al. · 2026-06-15

Read the paperPDF

Management
T3 Self-Evolving Memory
Feedback during deployment changes the memory-management policy itself.
Primary modality
Image
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. multi-branch clinical skill repository

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Reusable clinical skills are organized into general, task-level, and action-level branches and retrieved at episode onset through category routing plus semantic, utility, and temporal scoring.Full paper: §§3.2–3.3 · HTML full text · Figure 2; Equation 5

How memory changes (T2 operations)

  • Consolidation · Experience abstraction

    A gated buffer selects informative multimodal clinical trajectories and distills their reasoning or tool-use patterns into reusable skills with situational triggers and concrete steps.

  • Update · Prompted LLM/VLM record update

    The mutation pass selects CREATE, PATCH, or NONE and either writes a new skill or performs a local edit of an existing skill after novelty and quality review.

  • Revision · Evidence-weighted belief revision

    Window-level outcome advantages and positive, negative, or harmful adoption signals revise each skill's bounded utility estimate, which is subsequently used in retrieval.

  • Consolidation · Merging & deduplication

    Periodic repository governance merges redundant skills and keeps the three branches compact.

  • Forgetting · Discrete eviction

    Governance deprecates low-utility skills and removes the lowest-utility entries when a branch exceeds its capacity.

  • Forgetting · Progressive decay

    An explicit Ebbinghaus-style memory-strength term decays over time so recently reinforced skills outrank stale or rarely validated entries.

How the policy evolves (T3)

  • Retrieval policy

    In the online post-deployment stream, task outcomes and adoption feedback persistently PATCH skill content and situational triggers and revise utilities used by later category-aware retrieval; the updated repository is then read by subsequent clinical tasks.

Related systems