Multimodal Agent MemorySubmit a paper

VideoDirectorGPT

VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning

Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal · 2024-07-12

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Shared entity embeddings reused across scenes (consistency groupings)

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceRetained but not accessible to the agent
    Fused entity embeddings (CLIP image+text+bbox via small MLP); Per-scene layouts. The same entity embeddings are reused across shots as consistency groupings.Full-paper method / memory representation and retrieval · PDF p. 4 · Figure 1 / method overview

Related systems