Multimodal Agent MemorySubmit a paper

EgoMAS

MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents

Kangsan Kim, Yanlai Yang, Suji Kim, Woongyeong Yeo, Youngwan Lee, Mengye Ren, et al. · 2026-03-11

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Video & streaming
Also: text
Representation
Hybrid
The system-level 4W1H event records and the per-agent caption records are deliberately linked (events index into the agents whose captions produced them) and are jointly supplied to the answer generator through the two-stage retrieval cascade.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. per-agent individual memory: VLM captions of each embodied agent's egocentric video stream, produced at fixed 10-minute intervals over the multi-day recording

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceRetained but not accessible to the agent
    Each agent's long egocentric video is compressed into natural-language captions at every 10-minute interval; these per-agent caption records are the modality-native perception converted into a persistent textual store that the second retrieval stage queries. Each caption is tied to its agent identity and its 10-minute window, so the originating video segment remains locatable.5.1 Event-based Shared Memory · arXiv HTML full text, Section 5.1 · Figure 5 (EgoMAS pipeline overview)
    The second retrieval stage issues agent-specific sub-queries, each pairing an agent identifier with a sub-question, and retrieves top-k entries from that agent's individual caption memory; the agent identifier acts as a metadata routing key on top of lexical similarity ranking.5.2 Agent-wise Dynamic Retrieval · arXiv HTML full text, Section 5.2 · Table 5b (retrieval ablation, BM25 vs alternatives)

  2. system-level shared event memory: 4W1H (When, What, Where, Who, How) event records synthesised by a central manager from the parallel per-agent captions

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceRetained but not accessible to the agent
    A centralised manager integrates the individual captions into system-level summaries by identifying key events and explicitly recording, for each event, the fields When, What, Where, Who and How. The result is a schematised spatio-temporal record that aligns several agents' viewpoints into one coherent global memory, distinct from the flat per-agent caption stream it is built from.5.1 Event-based Shared Memory · arXiv HTML full text, Section 5.1 · Figure 5 (EgoMAS pipeline overview)
    Answering runs a two-tier read: BM25 ranking over the system-level event memory returns top-n events, which condition agent-specific sub-queries into the lower per-agent tier; a score threshold filters the union before response generation. The two tiers form one hierarchical store rather than independent indexes.5.2 Agent-wise Dynamic Retrieval · arXiv HTML full text, Section 5.2 · Figure 5

Related systems