Multimodal Agent MemorySubmit a paper

VideoStreaming

Streaming Long Video Understanding with Large Language Models

Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, et al. · 2024-05-25

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Image, text
Representation
Internalized latent & parametric

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. propagated video memory

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inFlat sequential records
    3. Retrieved bySimilarity search, Latent / parametric activation
    4. Source evidenceSource not retained
    Each clip is folded into constant-size historical latent memory.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Update · Learned update module

    A fixed learned update propagates memory across clips. For VideoStreaming, this specifically supports a trained but frozen update module of propagated video memory.

Related systems