Multimodal Agent MemorySubmit a paper

StreamChat

Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, et al. · 2025-01-23

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Image, text, dialogue
Representation
Hybrid
Caption clues retrieve the corresponding stored visual feature units, which are supplied together with short memory.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. short-term feature set and long-term feature-caption tree

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceRetained but not accessible to the agent
    Short memory retains selected vision embeddings; long memory clusters chunk features and captions into a chronological tree for cosine-similarity retrieval.Full paper: Memory · PDF p. 6 · Figure 5

  2. long-memory captions and encoded dialogue history

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource not retained
    Text clues index long-memory visual units, while prior question-answer pairs are encoded into a FAISS-indexed dialogue memory.Full paper: Memory · PDF p. 5

How memory changes (T2 operations)

  • Update · Rule-based update

    The memory-formation thread refreshes short memory, builds long-memory tree nodes, and appends dialogue records under a fixed schedule. For StreamChat, this specifically supports fixed-rule insertion or replacement of short-term feature set and long-term feature-caption tree.

  • Consolidation · Episodic / hierarchical summarization

    Chunk captions and higher-level grouped captions provide progressively compressed summaries. For StreamChat, this specifically supports episodic or hierarchical summarization of short-term feature set and long-term feature-caption tree.

  • Forgetting · Discrete eviction

    Age-dependent probabilities select a fixed number of embeddings for the new short-term set; unselected candidates leave the active memory. For StreamChat, this specifically supports explicit deletion or eviction of stored items of short-term feature set and long-term feature-caption tree.

Related systems