Multimodal Agent MemorySubmit a paper

TaskMem

Task-Focused Memorization for Multimodal Agents

Tao Zou, Yichen He, Tian Qiu, Yuan Lin, Hang Li · 2026-05-29

Read the paperPDF

Management
T3 Self-Evolving Memory
Feedback during deployment changes the memory-management policy itself.
Primary modality
Video & streaming
Also: Image, Audio & speech, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. temporally ordered event-centric textual memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource not retained
    For each streaming video segment, the memorization policy emits an episodic memory conditioned on recent segments and prior memories; evaluation answers using memory alone.Full-paper method / memory representation and retrieval · PDF p. 5

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During operation, TaskMem applies LLM-mediated memory editing to temporally ordered event-centric textual memory; the memory state changes while the governing update policy remains fixed.

How the policy evolves (T3)

  • Write policy

    After deployment, recent environment tasks define preference feedback used to update a lightweight adapter that persistently changes what the memorization policy writes. The persistent later-use effect is deployment-feedback evolution of the reusable write policy.

Related systems