Multimodal Agent MemorySubmit a paper

Task-driven Perception and Manipulation

Task-Driven Perception and Manipulation for Constrained Placement of Unknown Objects

Chaitanya Mitash, Rahul Shome, Bowen Wen, Abdeslam Boularias, Kostas E. Bekris · 2020-06-28

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, depth
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. dynamic object-centric voxel representation with occupied and unseen sets

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceRetained but not accessible to the agent
    The robot maintains an object-frame voxel representation split into observed occupied surface voxels and still-unseen voxels. Conservative and optimistic object volumes derived from these sets are queried to plan safe grasps, sensing viewpoints, handoffs and constrained placements.Full paper: Secs. III-IV Problem Setup and Proposed Pipeline · PDF pp. 3-4 · Figure 3

How memory changes (T2 operations)

  • Update · Rule-based update

    After each new RGB-D viewpoint, transformed observations add previously unseen surface voxels to the occupied set and remove voxels shown to be empty. Grasps and placements are then recomputed over the updated object estimate under a fixed update rule.

  • Revision · Rule-based state correction

    Pose tracking corrects the object pose after stochastic in-hand motion, and the new registered depth observation revises occupied, unseen and empty-space assignments before the next manipulation plan is selected.

Related systems