Multimodal Agent MemorySubmit a paper

MARS

MARS: Technical Report for the CASTLE Challenge at EgoVis 2026

Haoyu Zhang, Qiaohui Chu, Yisen Feng, Meng Liu, Weili Guan, Yaowei Wang, et al. · 2026-05-18

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: text, audio, Image
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. source-specific multimodal evidence memory

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval, Metadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    Videos, transcripts, gaze, heartrate, photos, and thermal imagery remain available as source-specific evidence.Full paper: 2.2. Offline Evidence Memory Construction · PDF p. 3 · Figure 1

  2. long-video captions and summaries

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Long videos are compressed into captions and summaries while other source modalities remain accessible.Full paper: Main pipeline · PDF p. 2 · Figure 1

Related systems