Multimodal Agent MemorySubmit a paper

LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics

LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics

Marc Glocker, Peter Hönig, Matthias Hirschmanner, Markus Vincze · 2025-04-30

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. QA-style records of past interactions

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval, Structure traversal
    4. Source evidenceSource not retained
    Past interactions are converted into textual memory entries.; Entries are embedded and appended to a vector database.; Object detections and user/robot interaction records become searchable knowledge-base items.. graph query/traversal over the memory conditioned on the current goal.Full paper: B. Algorithmic Framework · PDF p. 3 · Figure 1 / method overview

  2. Object identities and last-known locations

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval, Structure traversal
    4. Source evidenceSource not retained
    Past interactions are converted into textual memory entries.; Entries are embedded and appended to a vector database.; Object detections and user/robot interaction records become searchable knowledge-base items.. graph query/traversal over the memory conditioned on the current goal.Full paper: B. Algorithmic Framework · PDF p. 3 · Figure 1 / method overview

  3. Action history and task state

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval, Structure traversal
    4. Source evidenceSource not retained
    Past interactions are converted into textual memory entries.; Entries are embedded and appended to a vector database.; Object detections and user/robot interaction records become searchable knowledge-base items.. graph query/traversal over the memory conditioned on the current goal.Full paper: B. Algorithmic Framework · PDF p. 3 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics applies fixed-rule insertion or replacement to QA-style records of past interactions; the memory state changes while the governing update policy remains fixed.

Related systems