Multimodal Agent MemorySubmit a paper

ReMA (Recursive Multimodal Agent)

Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline

Guo Chen, Lidong Lu, Yicheng Liu, Liangrui Dong, Lidong Zou, Jixin Lv, et al. · 2026-03-05

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: audio, text
Representation
Hybrid
The language-augmented Memory Bank and the timestamp-addressable raw stream are deliberately linked: memory nodes carry the temporal intervals that let the controller route back to MMInspect for fresh perceptual evidence, and both routes are jointly supplied to the controller's recursive belief update.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Memory Bank B of consolidated linguistic summaries produced by MMInspect over sampled video segments

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    MMInspect samples frames inside a requested time range and calls a VLM to produce localized descriptions; those observations are incrementally consolidated into the Memory Bank B by MemManage, giving a compact global language-augmented representation of the lifelong stream. Memory nodes are keyed by their source temporal intervals, so MemSearch first ranks relevant nodes and then hierarchically summarizes across temporal intervals, and the controller can always re-issue MMInspect on a stored interval to recover the original frames. Memory content is text derived from perception, not pixels.Full paper (arXiv HTML): Sec. 5 Method; Algorithm 1 (controller loop), Algorithm 2 (MMInspect), Algorithm 3 (MemManage), Algorithm 4 (MemSearch) · Figure 6 (ReMA two-phase overview); Algorithms 1-4 (Appendix B.1)

  2. the retained lifelong video/audio stream itself, re-inspectable by interval through the MMInspect tool

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource directly available
    At every reasoning step the controller may choose the MMInspect action to re-examine a specific time interval of the raw stream, sampling frames afresh for either passive captioning or query-based inspection. The source stream is therefore retained as an operative, re-queryable memory route addressed by timestamp, not merely consumed once at ingest.Full paper (arXiv HTML): Sec. 5 Method; Algorithm 1 lines 10-24; Algorithm 2 (MMInspect) · Figure 6

How memory changes (T2 operations)

  • Consolidation · Merging & deduplication

    When new observations arrive during inference, MemManage detects overlapping memory nodes and merges them by summarization, explicitly to prevent memory explosion. Memory content changes at inference time under a fixed, hand-specified consolidation rule.

  • Consolidation · Episodic / hierarchical summarization

    New MMInspect observations are incrementally consolidated into B by summarization, and MemSearch performs hierarchical summarization across temporal intervals; the recursive belief state is progressively refined by folding each step's outcome back into the persistent Memory Bank.

  • Update · Prompted LLM/VLM record update

    The LLM/VLM controller writes the outcome of each MMInspect or MemSearch action back into B, so the belief state is edited by model-generated content during the inference episode; the governing update policy (the fixed controller prompt and MemManage rule) does not change.

Related systems