Multimodal Agent MemorySubmit a paper

PEAM

PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft

Yuchen Guo, Junli Gong, Weicheng Wang, Hongmin Cai, Yiu-ming Cheung, Weifeng Su · 2026-06-01

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Hybrid
The explicit episodic trajectory store and the parametric adapter bank form one deliberate pipeline: staged symbolic trajectories are selected and consolidated into the parametric memory that then drives the fast execution path.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. per-category MoE-LoRA adapter bank over the multimodal fast module

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Experience is internalized into a multimodal Mixture-of-Experts LoRA over a Qwen3-VL-8B-Instruct backbone with physically isolated per-category adapters for {craft, gather, combat}; consolidation updates only the category-specific adapter parameters via a joint behavioral-cloning plus PEAM-DPO contrastive objective on success and failure-correction pairs. The stored memory is the adapter weights themselves; no source trajectory is recoverable from them.3.1 Two-tier architecture; 3.2 Contrastive internalization · Figure 2; Equation 1
    At execution time a task is routed to its category adapter and the fast path emits the consolidated skill reflexively from the visually conditioned parametric module; the slow LLM is invoked only when verification fails. Activation of memory is thus parametric rather than by explicit lookup.4.1 Execution loop · Algorithm 1

  2. external episodic store of successful and failure-correction trajectories

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource not retained
    An external episodic store stages tuples (task, skill category, executed trajectory, outcome) accumulated during live agent execution. Entries are executable JavaScript action code plus a discrete pre-action state vector (inventory composition, biome identity, time-of-day bucket, bot location); candidates are selected for consolidation by matching that state vector and by a Parameterization Worthiness score over cost saving, cross-context stability, redundancy, and interference. Raw visual observations are not retained, so the perceptual source is not recoverable.3.2 Contrastive internalization; 3.3 Parameterization Worthiness · Algorithm 1 line 17; Equation 3; Appendix D

How memory changes (T2 operations)

  • Update · Optimization-based update

    During the agent's live Minecraft operation, a scale-free statistical trigger compares each skill's recent failure rate against its own historical baseline and, when the criterion fires, runs a gradient update of the category adapter over the staged trajectories. The parametric memory content therefore changes at deployment time; the trigger rule and the PV selection formula that govern the update are fixed.

  • Consolidation · Experience abstraction

    Batches of concrete staged trajectories, including failure-correction contrasts, are abstracted into reusable parametric skills inside the adapter bank, replacing inference-time retrieval of individual episodes; later tasks in the same category reuse the abstracted skill.

Related systems