Multimodal Agent MemorySubmit a paper

SkillMimicGen

SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment

Caelan Reed Garrett, Ajay Mandlekar, Bowen Wen, Dieter Fox · 2024-10-24

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Embodied & 3D
Also: Image, proprioception, actions
Representation
Internalized latent & parametric

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. fixed bank of learned parameterized manipulation skills with initiation conditions, closed-loop controllers, and termination conditions

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank
    3. Retrieved byPlanner / tool-assisted retrieval, Latent / parametric activation
    4. Source evidenceSource not retained
    A skill is represented as an object, an initiation condition, a closed-loop controller, and a termination condition. At test time SkillGen executes several of these learned skills in sequence, with motion planning between them, so the learned skill modules remain available as fixed operative artifacts during later robot execution.Full paper: Sec. 4.1 Skills Framework · PDF p. 4 · Figure 2
    Generated demonstrations are used to learn parameterized skill policies. During deployment, the system predicts a skill initiation state, plans to it, rolls out the corresponding learned policy until its termination condition, and repeats this process for the next skill.Full paper: Sec. 4.6 Policy Learning · PDF p. 5 · Skill definition and deployment procedure
    The deployed agents receive front-view and wrist-camera RGB observations together with proprioception, establishing direct non-text participation in skill selection and execution.Full paper: Sec. 5 Experimental Setup · PDF p. 6

Related systems