Multimodal Agent MemorySubmit a paper

MindForge

MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Cultural Learning

Mircea Lica, Ojas Shirekar, Baptiste Colle, Chirag Raman · 2025-01-01

Read the paper

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Structured theory-of-mind state linking percepts, beliefs, desires, and actions

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval, Structure traversal
    4. Source evidenceSource not retained
    Agents accumulate cultural/task experiences during Minecraft play.; Communication and observations are written into memory components and ToM state.; The framework stores information useful for later adaptation, collaboration, and transfer.. spatial query over the map conditioned on current observation and instruction.Full paper: Memory · PDF p. 4 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During operation, MindForge applies LLM-mediated memory editing to Structured theory-of-mind state linking percepts, beliefs, desires, and actions; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Experience abstraction

    During operation, MindForge applies abstraction of episodes into reusable experience to Structured theory-of-mind state linking percepts, beliefs, desires, and actions; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Episodic / hierarchical summarization

    During operation, MindForge applies episodic or hierarchical summarization to Structured theory-of-mind state linking percepts, beliefs, desires, and actions; the memory state changes while the governing update policy remains fixed.

Related systems