Multimodal Agent MemorySubmit a paper

VideoAgent

VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding

Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, et al. · 2024-03-18

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: Image, text
Representation
Hybrid
Source-linked captions, visual features, relational object state, and recoverable video segments are jointly routed by tools.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. temporal memory

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records, Vector-indexed collection
    3. Retrieved byMetadata / spatiotemporal retrieval, Similarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Segment captions, caption embeddings, and time positions form temporal memory; tools retrieve captions by time or semantic relevance.Full-paper method / memory representation and retrieval · PDF p. 6 · Figure 2

  2. temporal memory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Each video segment has a visual feature used for query-to-segment localization, with the segment identity retained.Full-paper method / memory representation and retrieval · PDF p. 3 · Figure 2

  3. object memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Re-identified objects are stored in a feature table and SQL database with categories and temporal appearances; an LLM tool generates database queries.Full-paper method / memory representation and retrieval · PDF p. 6 · Figure 2

  4. raw video available to VQA tool

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource directly available
    After localizing a segment from memory, the agent can invoke visual question answering on the corresponding raw video segment.Full-paper method / memory representation and retrieval · PDF p. 3 · Figure 2

Related systems