Multimodal Agent MemorySubmit a paper

LUMA-RAG

LUMA-RAG: Lifelong Multimodal Agents with Provably Stable Streaming Alignment

Rohan Wandre, Yash Gajewar, Namrata Patel, Vivek Dhalkari · 2025-11-04

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text, Image, audio
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. multimodal embeddings

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection, Hierarchical multi-tier store
    3. Retrieved bySimilarity search
    4. Source evidenceSource not retained
    Hot HNSW memory tier; compressed IVFPQ memory tier; CLAP-to-CLIP alignment bridge. cross-modal retrieval over aligned embedding spaces, including text-to-image and audio-to-image.Full paper: C. Multi-Tier Memory and Policy · PDF p. 2 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Optimization-based update

    During operation, LUMA-RAG applies a fixed operational optimization objective to multimodal embeddings; the memory state changes while the governing update policy remains fixed.

  • Reorganization · Memory-level migration (short- to long-term)

    Incremental orthogonal Procrustes updates maintain CLAP-to-CLIP alignment as streams evolve. For LUMA-RAG, this specifically supports migration between memory tiers of multimodal embeddings.

  • Update · Rule-based update

    During operation, LUMA-RAG applies fixed-rule insertion or replacement to multimodal embeddings; the memory state changes while the governing update policy remains fixed.

Related systems