Multimodal Agent MemorySubmit a paper

VideoTree

VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos

Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, et al. · 2025-03-14

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: Image, text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. query-adaptive hierarchical tree over video clusters, whose nodes hold a representative keyframe and its VLM caption

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byStructure traversal, Metadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Frames are encoded with EVA-CLIP-8B and K-Means clustered; the frame nearest each centroid becomes the cluster keyframe and is captioned by a VLM; the LLM scores each cluster caption for query relevance on a 1-3 scale, and the cluster count k is doubled and the loop repeated until enough high-relevance information is present or a breadth cap is hit.3.1 Adaptive breadth expansion · Figure 2; Algorithm 1 lines 3-12
    High-relevance first-level clusters are re-clustered into sub-clusters, giving the tree depth: 'somewhat relevant' clusters receive w sub-clusters and 'highly relevant' clusters receive a two-level subtree of branch width w, so keyframe budget is allocated hierarchically by relevance.3.2 Relevance-guided depth expansion · Figure 2
    The tree is traversed to collect the selected node captions, which are re-ordered by their original timestamps and concatenated into a video description that is fed to the LLM for answering. Each caption stays tied to its source keyframe index, so the underlying frames remain recoverable from the node.3.3 LLM reasoning over the tree · Algorithm 1 lines 17-18

Related systems