Multimodal Agent MemorySubmit a paper

REVEAL (Retrieval-Augmented Visual-Language Pre-Training)

REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, et al. · 2023-04-03

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Image
Also: text
Representation
Internalized latent & parametric

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. large-scale multi-source knowledge memory of compressed value embeddings (WIT image-text passages, CC12M web images with alt-text, VQA-v2 pairs, WikiData triplets)

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceRetained but not accessible to the agent
    Every knowledge item -- including raw image-text pairs -- is run through a single unified encoder that emits (i) one d-dimensional key vector used only for MIPS lookup and (ii) a value embedding compressed by a Perceiver to c=32 latent tokens. The compressed value token sequences are the persisted artifact: at query time the top-K (K=10 pre-training, K=50 fine-tuning) value token sequences are concatenated to the query and consumed directly by the generator through an attentive-fusion cross-attention module whose attention mask carries the retrieval score. The stored representation is therefore a persistent multimodal latent state consumed downstream, not merely a retrieval index; the key embedding alone would not constitute an independent path.Full paper: Sec. 3.2 Memory (unified encoder + Perceiver compression); Sec. 3.3 Retriever; Sec. 3.4 Generator with attentive fusion; Sec. 4.2 knowledge sources; Sec. 4.3 indexing · CVPR/arXiv PDF pp. 3-5 · Figure 2(b) memory value compression; Figure 2(c) retriever; Figure 3 attentive fusion; Table 1 knowledge-source statistics

Related systems