Multimodal Agent MemorySubmit a paper

Visual Memory QA

Visual Memory QA: Your Personal Photo and Video Search Agent

Lu Jiang, LiangLiang Cao, Yannis Kalantidis, Sachin Farfade, Alex Hauptmann · 2017-01-01

Read the paper

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Image
Also: text, video
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. automatically analyzed personal visual memories

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Content-analysis outputs and indexed visual evidence without user-generated metadata. conversational interface retrieves relevant personal visual evidence to answer life-log questions.Full paper: Visual Memory Question Answering · PDF p. 1 · Figure 1 / method overview

  2. detected visual concepts/events

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource not retained
    Content-analysis outputs and indexed visual evidence without user-generated metadata. conversational interface retrieves relevant personal visual evidence to answer life-log questions.Full paper: Visual Memory Question Answering · PDF p. 1 · Figure 1 / method overview

Related systems