Multimodal Agent MemorySubmit a paper

MMed-RAG

MMed-RAG: Versatile Multimodal RAG System for Medical Vision-Language Models

Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, et al. · 2025-03-03

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Image
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. domain-partitioned clinical-report knowledge base (radiology, ophthalmology, pathology report corpora built from dataset training splits), indexed by an image-text contrastive multimodal retriever

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource directly available
    The persistent store is a corpus of medical reports drawn from the training splits of five datasets, partitioned by medical domain. A BiomedCLIP-based domain-identification module labels the incoming medical image, which selects the corresponding domain-specific multimodal retriever; that retriever (ResNet-50 image encoder + BioClinicalBERT text encoder trained contrastively to align an image with its own report) embeds the query image and does similarity search over the report embeddings, returning top-k reports that are appended verbatim as context for the Med-LVLM. The stored items are natural-language report text, retrieved directly and read verbatim by the generator.Full paper: Sec. 3.1 Domain-Aware Retrieval Mechanism; Sec. 3.2 Adaptive Retrieved Context Selection · arXiv HTML, method section · Algorithm 1 (retrieval step, line 23); Figure 2 system overview

Related systems