Multimodal Agent MemorySubmit a paper

VisRAG

VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents

Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, et al. · 2024-10-14

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Image
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. visual document corpus

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource directly available
    Document pages remain in image form; a VLM encodes each page for cosine-similarity retrieval.Full-paper method / memory representation and retrieval · PDF p. 5 · Figure 2
    Retrieved page images, rather than their index embeddings, are supplied to the VLM generator.Full-paper method / memory representation and retrieval · PDF p. 4

Related systems