Multimodal Agent MemorySubmit a paper

Continual Harness

Continual Harness: Online Adaptation for Self-Improving Foundation Agents

Seth Karten, Joel Zhang, Tersoo Upaa Jr, Ruirong Feng, Wenzhe Li, Chengshuai Shi, et al. · 2026-05-11

Read the paperPDF

Management
T3 Self-Evolving Memory
Feedback during deployment changes the memory-management policy itself.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Hybrid
Textual strategy memory and executable skill procedures are jointly supplied to the online agent.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. persistent textual facts and strategy memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Rendered frames, textual game state, actions, and outcomes are summarized into persistent harness memory that the agent consults during later embodied gameplay.Full paper: §§2.2–3.1 · HTML full text · Figure 2

  2. persistent executable skills and subagent procedures

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inModular adapter bank
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource not retained
    The harness exposes editable skills and subagent procedures whose descriptions and executable bodies are revised from recent multimodal trajectory failures and invoked on later steps.Full paper: §§2.2–3.1; Appendix · HTML full text · Figure 2

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    At fixed online intervals, the Refiner applies create, read, update, and delete edits to the persistent harness prompt, subagents, skills, and memory.

  • Consolidation · Experience abstraction

    Recent gameplay trajectories are summarized into failure signatures and reusable behavioral guidance before the harness is revised.

How the policy evolves (T3)

  • Retrieval policy

    Every fixed number of online steps, feedback over recent rendered-frame trajectories revises persistent skill descriptions and procedures in place; the agent immediately retrieves and executes the revised harness on subsequent steps without reset.

Related systems