Multimodal Agent MemorySubmit a paper

Pensieve

Memory-QA: Answering Recall Questions Based on Multimodal Memories

Hongda Jiang, Xinyuan Zhang, Siddhant Garg, Rishab Arora, Shiun-Zu Kuo, Jiayang Xu, et al. · 2025-09-22

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Image
Also: text, time, location
Representation
Hybrid
Text and structured metadata remain linked to the original image entry; the VLM configuration can directly consume retrieved images.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. multimodal memory entries

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    Each stored memory entry contains an image snapshot plus its invocation, timestamp, and location metadata.Full paper: Memory Retrieval · PDF p. 4

  2. offline memory augmentation

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    OCR, QA-guided visual descriptions, and completed invocation text augment each source memory before retrieval.Full paper: Memory Retrieval · PDF p. 4 · Figure 2

  3. memory metadata

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Date match, recency, location similarity, and multimodal embedding similarity are fused to rank memory entries.Full-paper method / memory representation and retrieval · PDF p. 5 · Figure 3

Related systems