Multimodal Agent MemorySubmit a paper

HIMM

HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering

Ji Li, Bo Wang, Jing Xia, Mingyi Li, Shiyan Hu · 2026-03-02

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. episodic + semantic memory

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Latent / parametric activation
    4. Source evidenceSource directly available
    Stores global/regional embeddings, 3D boxes, raw images and episodic traces. retrieves similar episodes and semantic regions for exploration/QA.Full paper: Episodic Memory: Visual Reasoning for · PDF p. 3 · Figure 1 / method overview

  2. episodic + semantic memory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Latent / parametric activation
    4. Source evidenceRetained but not accessible to the agent
    Stores global/regional embeddings, 3D boxes, raw images and episodic traces. retrieves similar episodes and semantic regions for exploration/QA.Full paper: Episodic Memory: Visual Reasoning for · PDF p. 3 · Figure 1 / method overview

  3. episodic + semantic memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Latent / parametric activation
    4. Source evidenceSource not retained
    Stores global/regional embeddings, 3D boxes, raw images and episodic traces. retrieves similar episodes and semantic regions for exploration/QA.Full paper: Episodic Memory: Visual Reasoning for · PDF p. 3 · Figure 1 / method overview

  4. episodic + semantic memory

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank, Vector-indexed collection
    3. Retrieved bySimilarity search, Latent / parametric activation
    4. Source evidenceSource not retained
    Stores global/regional embeddings, 3D boxes, raw images and episodic traces. retrieves similar episodes and semantic regions for exploration/QA.Full paper: Episodic Memory: Visual Reasoning for · PDF p. 3 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, HIMM applies fixed-rule insertion or replacement to episodic + semantic memory; the memory state changes while the governing update policy remains fixed.

Related systems