Multimodal Agent MemorySubmit a paper

Train the Agent Not the Expert

Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning

Yaowu Fan, Tao Han, Dazhao Du, Andy J. Ma, Jia Wan · 2026-05-28

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. dynamic record of expert calls, intermediate visual evidence, and agent reasoning state across turns

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Heterogeneous-expert call records plus intermediate visual evidence held across multi-turn reasoning. experts are selected and their outputs integrated; evidence is referenced across turns.Full-paper method / memory representation and retrieval · PDF p. 9 · Figure 1 / method overview

How memory changes (T2 operations)

  • Reorganization · Memory-level migration (short- to long-term)

    During operation, Train the Agent Not the Expert applies migration between memory tiers to dynamic record of expert calls, intermediate visual evidence, and agent reasoning state across turns; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Episodic / hierarchical summarization

    During operation, Train the Agent Not the Expert applies episodic or hierarchical summarization to dynamic record of expert calls, intermediate visual evidence, and agent reasoning state across turns; the memory state changes while the governing update policy remains fixed.

  • Update · Rule-based update

    During operation, Train the Agent Not the Expert applies fixed-rule insertion or replacement to dynamic record of expert calls, intermediate visual evidence, and agent reasoning state across turns; the memory state changes while the governing update policy remains fixed.

Related systems