Multimodal Agent MemorySubmit a paper

ReKV

Streaming Video Question-Answering with In-context Video KV-Cache Retrieval

Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, et al. · 2025-03-01

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. external video KV-Cache store built incrementally from the live video stream and queried per question

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search
    4. Source evidenceSource not retained
    Frames are encoded once with sliding-window attention and their per-layer key/value vectors are kept as an external memory: 'All video KV-Caches are stored for future retrieval.' The store is tiered across GPU, RAM and disk (a 1-hour video yields 18.8 GB of cache for LLaVA-OV-7B). At question time the r most relevant frames/blocks are selected either by external CLIP-style cosine similarity between frame and question embeddings (Eq. 2) or by internal similarity between per-frame averaged attention keys and aggregated question queries, and only those KVs are reloaded onto the GPU as context for generation (Eq. 3). Frame pixels are not retained in the memory.Full paper: Sec. 3 (video stream encoding with sliding-window attention; video KV-Cache retrieval; question answering with retrieved KVs); Appendix A.1, A.3 · HTML Sec. 3; Appendix A.1/A.3 · Figure 2 (a)-(c); Eq. 1-3

How memory changes (T2 operations)

  • Update · Rule-based update

    The persistent KV memory is written during deployment, not before it: as the stream advances, each newly encoded frame's key/value vectors are appended to the store under a fixed sliding-window encoding rule, and questions may be asked at any point against the store as it stands. The memory content therefore changes across inference while the write rule is fixed.

  • Reorganization · Memory-level migration (short- to long-term)

    'For extremely long videos, we manage memory constraints by offloading KV-Caches to RAM or disk'; Appendix A.1 adds that video processing stores the computed cache in RAM and spills to disk when RAM is exceeded. On retrieval the selected entries are 'reloaded onto the GPU', so entries move between storage tiers during streaming under a fixed capacity rule.

Related systems