Multimodal Agent MemorySubmit a paper

OmAgent

OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer

Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, Kyusong Lee · 2024-11-12

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: Image, text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. retrievable video frames

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval, Metadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    OmAgent indexes frames and retrieves them through an autonomous divide-and-conquer loop.Full paper: Method · PDF p. 3 · Figure 1
    Before semantic retrieval, OmAgent filters the knowledge database by timestamps generated during divide-and-conquer localization.Full paper: Method · PDF p. 3 · Figure 1

Related systems