Multimodal Agent MemorySubmit a paper

HiMe

HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control

Li Ji, Siyin Wang, Pengfang Qian, Xiaopeng Yu, Yihai Tian, Zhaoye Fei, et al. · 2026-07-03

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Hybrid
Retained RGB frames and their textual captions/tags are stored in one key-value record and jointly supplied to the Planner at re-planning time; the procedural subgoal plan is revised in the same call, so the paths are deliberately fused rather than parallel alternatives.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. episodic contextual memory (multimodal key-value store) - retained RGB observation frames referenced by image_path and re-supplied to the Planner

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource directly available
    Contextual memory E^c_t is a multimodal key-value store M_t accumulating 'both visual imagery and textual descriptions'. Records carry id, tags, data and an image_path pointing at the retained dual-view RGB frame; retrieved records, including the stored imagery, are handed back to the Planner (composite wrist + third-person view) at re-planning time, so the raw perceptual evidence itself persists and is reused downstream rather than only a textual proxy.Full paper: Sec. 3.2 Hierarchical Embodied Memory · HTML v1, Sec. 3.2 (approx. PDF pp. 7-8) · Figure 2 (Executor/Sentry/Planner loop); Eq. (3)
    Image-only, text-only and cross-modal memory variants are compared; the cross-modal store (image plus text) is best (avg. 90%), confirming that retained imagery is an operative memory path and not decoration.Full paper: Sec. 5.1 (Q1: What type of memory representation is most effective) · HTML v1, Sec. 5.1 · ablation over memory representation

  2. episodic contextual memory - textual captions/tags distilled from observations, used as the retrieval keys and as Planner context

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource linked & recoverable
    Each memory record stores tags and a textual description of what was observed. A query q_t = f_enc(l, W_t) built from the instruction and the recent-observation window is embedded with OpenAI text-embedding-3 and matched by cosine similarity against caption embeddings, returning M_ret = TopK(E^c_t, q_t). The textual record is the indexed surface, and it carries an image_path back to the source frame, so the originating observation stays recoverable.Full paper: Sec. 3.2 Hierarchical Embodied Memory · HTML v1, Sec. 3.2 (approx. PDF pp. 7-8) · Eq. (3)

  3. episodic procedural memory - ordered list of language subgoals with done/current/pending status labels

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Procedural memory E^p_t = {tau_i} is an ordered language-subgoal plan whose entries are labelled [done]/[current]/[pending]. It persists across subtasks, is read in full by the Planner when re-planning is triggered, and the next active subgoal tau_{t+1} is selected from it and dispatched to the Executor.Full paper: Sec. 3.2 / Sec. 3.3 (policy decomposition) · HTML v1, Sec. 3.2-3.3 (approx. PDF pp. 7-9) · Eq. (4); Sec. 10 Planner prompt (plan_list)

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During deployment, whenever the Sentry emits a subtask-completion signal u_t = 1 (Eq. 2), the Planner is invoked and executes (E^c_{t+1}, E^p_{t+1}) <- pi_p(l, W_t, M_ret, E^p_t): a GPT-4o Planner issues explicit CREATE / UPDATE / DELETE / QUERY operations over the multimodal record store and rewrites the subgoal plan. Memory content changes at inference time; the CRUD protocol and prompt that govern it are fixed.

  • Revision · LLM-based conflict resolution

    Update revises outdated entries and Delete removes 'stale or conflicting information'; the ablation shows the agent self-corrects its stored knowledge when the environment changes mid-episode or when a human instruction contradicts an earlier recorded preference. Resolution is performed by the LLM Planner reading the retrieved records against fresh observations.

  • Forgetting · Discrete eviction

    DELETE record_id removes obsolete records outright during deployment; the ablation notes that without Update/Delete the store retains obsolete states and injects noise, so eviction is an actually-exercised inference-time operation.

Related systems