Multimodal Agent MemorySubmit a paper

Affordance RAG

Affordance RAG: Hierarchical Multimodal Retrieval With Affordance-Aware Embodied Memory for Mobile Manipulation

Ryosuke Korekata, Quanting Xie, Yonatan Bisk, Komei Sugiura · 2026-01-01

Read the paper

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Embodied & 3D
Also: text, Image
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Target object candidates and receptacle candidates

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection, Hierarchical multi-tier store
    3. Retrieved byStructure traversal
    4. Source evidenceSource directly available
    The robot builds Affordance Mem during pre-exploration from collected images.; SEEM-style segmentation produces object masks; VLMs produce descriptions; embeddings are stored for multimodal retrieval.; Agglomerative clustering builds higher-level region nodes and summaries, making the memory hierarchical rather than a flat image bank.. graph query/traversal over the memory conditioned on the current goal.Full paper: A. Affordance-Aware Embodied Memory · PDF p. 3 · Figure 1 / method overview

  2. Hierarchical region nodes spanning building/room/area/object levels

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection, Hierarchical multi-tier store
    3. Retrieved byStructure traversal
    4. Source evidenceRetained but not accessible to the agent
    The robot builds Affordance Mem during pre-exploration from collected images.; SEEM-style segmentation produces object masks; VLMs produce descriptions; embeddings are stored for multimodal retrieval.; Agglomerative clustering builds higher-level region nodes and summaries, making the memory hierarchical rather than a flat image bank.. graph query/traversal over the memory conditioned on the current goal.Full paper: A. Affordance-Aware Embodied Memory · PDF p. 3 · Figure 1 / method overview

  3. Object-centric descriptions beyond category labels

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection, Hierarchical multi-tier store
    3. Retrieved byStructure traversal
    4. Source evidenceSource not retained
    The robot builds Affordance Mem during pre-exploration from collected images.; SEEM-style segmentation produces object masks; VLMs produce descriptions; embeddings are stored for multimodal retrieval.; Agglomerative clustering builds higher-level region nodes and summaries, making the memory hierarchical rather than a flat image bank.. graph query/traversal over the memory conditioned on the current goal.Full paper: A. Affordance-Aware Embodied Memory · PDF p. 3 · Figure 1 / method overview

  4. Affordance scores indicating whether picking/placing is physically suitable

    1. Represented asExperience-conditioned parametric memoryInternalized latent & parametric
    2. Stored inModular adapter bank, Vector-indexed collection
    3. Retrieved byStructure traversal, Latent / parametric activation
    4. Source evidenceSource not retained
    The robot builds Affordance Mem during pre-exploration from collected images.; SEEM-style segmentation produces object masks; VLMs produce descriptions; embeddings are stored for multimodal retrieval.; Agglomerative clustering builds higher-level region nodes and summaries, making the memory hierarchical rather than a flat image bank.. graph query/traversal over the memory conditioned on the current goal.Full paper: A. Affordance-Aware Embodied Memory · PDF p. 3 · Figure 1 / method overview

Related systems