Sensorimotor Primitives
Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations
Junchi Liang, Bowen Wen, Kostas Bekris, Abdeslam Boularias · 2022-03-08
- Management
- T2 Dynamic Memory
- Memory changes during deployment through a stable management policy.
- Representation
- Internalized latent & parametric
How experience becomes memory
Each path is one memory component, traced from what is stored to how it is found again.
LSTM memory of completed tool-target-waypoint choices and manipulation subtasks
- Represented asPersistent multimodal latent stateInternalized latent & parametric
- Stored inFlat sequential records
- Retrieved byLatent / parametric activation
- Source evidenceSource not retained
The paper states that the current 6-DoF object poses alone are insufficient because the robot must remember prior actions such as whether a brush was dipped. An LSTM compresses the history of selected tools, targets, and waypoints, and the high- and intermediate-level policies read this memory when choosing the next manipulation step.Full paper: Sec. III Problem Formulation and Architecture · PDF pp. 2-3 · Figure 2
The memory output encodes which subtasks in the demonstrated sequence have already been performed. The memory-consistency and retrieval objectives train it to retain and recover earlier target, tool, and waypoint choices for later decisions.Full paper: Sec. IV Learning from Visual Demonstrations · PDF p. 5 · Memory and retrieval losses
How memory changes (T2 operations)
Update · Learned update module
During a manipulation episode, the fixed learned LSTM update transforms the accumulated tool-target-waypoint history after each decision into a new persistent hidden state. Later policy decisions consume that changed state, while the update rule itself remains fixed.