Multimodal Agent MemorySubmit a paper

H2-EMV

Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deployment

Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour · 2026-05-05

Read the paperPDF

Management
T3 Self-Evolving Memory
Feedback during deployment changes the memory-management policy itself.
Primary modality
Embodied & 3D
Also: Image, audio, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. hierarchical scene instants, events and summaries

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Online construction builds a history tree from scene instants to long-range summaries.Full paper: Task and Architecture Overview · PDF pp. 13, 18, 21 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Episodic / hierarchical summarization

    Lower-level events are summarized upward. For H2-EMV, this specifically supports episodic or hierarchical summarization of hierarchical scene instants, events and summaries.

  • Forgetting · Progressive decay

    Tiered expiry and relevance remove detail while keeping placeholders. For H2-EMV, this specifically supports progressive decay of stored importance or retention of hierarchical scene instants, events and summaries.

How the policy evolves (T3)

  • Maintenance policy

    User feedback updates natural-language retention rules used on later experience. The persistent later-use effect is deployment-feedback evolution of the reusable maintenance policy.

Related systems