Multimodal Agent MemorySubmit a paper

UniversalRAG

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang · 2026-06-12

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: Image, text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. video corpus held at two granularities (full-length videos and segmented short clips)

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Videos from LVBench and the VideoRAG corpora are stored as full-length items and additionally segmented into short clips, forming two separate granularity-level corpora, each embedded with a vision encoder (InternVideo2 in v1, VLM2Vec-V2 in v5) into its own independent embedding space and read by cosine similarity.Full paper: Sec. 2.2 UniversalRAG; Sec. 3.1 Experimental Setup / Implementation Details · Figure 1 (framework overview); Figure 4 (routed-modality distribution)
    32 frames are uniformly sampled per video for both the retrieval and the generation stage, so the retrieved video is handed to the LVLM generator as decoded frames rather than as a caption; the stored perceptual source is therefore directly accessible downstream. Scripts (for videos) and captions (for images) enter only the ensemble similarity score used for retrieval, not as a substitute memory record.Full paper: Sec. 3.1 Implementation Details (v1 wording retained across versions)

  2. image corpus kept intact (MRAG-Bench / WebQA / InfoSeek images)

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Images are kept intact because they are inherently piecemeal, indexed in their own vision embedding space, and the retrieved image itself is supplied to the multimodal generator (Qwen3-VL-8B-Instruct / InternVL3.5-8B / Molmo2-4B), which consumes textual and visual inputs.Full paper: Sec. 2.2 UniversalRAG; Sec. 3.1 Experimental Setup · Figure 1
    Access is gated by a per-query modality-and-granularity router that runs before retrieval and selects which corpus (none / paragraph / document / table / image / clip / video) to query, i.e. a planner-style routing step layered on top of within-corpus similarity search.Full paper: Sec. 2.2 / Sec. 2.3 Router Implementation Strategies · Figure 1

Related systems