Multimodal Agent MemorySubmit a paper

TeleMem

TeleMem: Building Long-Term and Multimodal Memory for Agentic AI

Chunliang Chen, Ming Guan, Xiao Lin, Luxi Lin, Jiaxu Li, Qiyi Wang, et al. · 2026-01-22

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: Image, text
Representation
Hybrid
The semantic DAG and the retained video material are deliberately linked: event and object nodes carry clip references and timestamps that the video tools dereference, and the ReAct agent composes graph-retrieved textual context with clip-level visual answers into one response.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Threaded directed acyclic graph of stabilized semantic states: profile states, event states summarizing dialogue and video clips, and entity/object states with temporal evolution

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Vector-indexed collection
    3. Retrieved bySimilarity search, Structure traversal, Metadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Memory is a directed acyclic graph whose nodes are stabilized semantic states of three kinds: profile states over user and bot attributes, event states capturing interaction semantics drawn from textual dialogue or from video clips, and entity/object states representing persistent multimodal entities and their temporal evolution. Each node carries consolidated content, an embedding for similarity search, and effective timestamps, with the temporal constraint tau(p) < tau(v_i) enforcing acyclicity. Edges are pruned so only minimal causal dependencies survive, yielding interpretable memory threads as root-to-node paths. Retrieval identifies seed nodes by embedding similarity and then performs closure-based backward dependency traversal followed by chronological linearization, so the graph structure and node timestamps are both operative at read time. Serialized nodes (Appendix A, Figures 5-6) carry textual captions, aligned timestamps and clip references, linking a node back to the video segment it summarizes.Full paper (arXiv HTML v4): Sec. 3.1 / 3.1.1 memory representation; Sec. 3.2 graph maintenance; Appendix A node examples · HTML Sec. 3.1-3.2, Appendix A · Figures 5-6 (serialized node examples)

  2. Retained video material addressed by the memory agent's video tools (video.retrieval for clip timestamps, video.rag for textual aggregation, video.qa for visual question answering over clips)

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval, Metadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    A ReAct-style agent answers multimodal queries by invoking three tools over the stored video material: video.retrieval returns clip timestamps, video.rag aggregates textual evidence, and video.qa performs visual question answering directly on the clips. Because video.qa answers by looking at clips at retrieval time rather than at pre-extracted captions, raw visual evidence remains reachable from memory rather than being discarded after summarization.Full paper (arXiv HTML v4): Sec. 3.3.2 multimodal memory agent; Algorithm 1 · HTML Sec. 3.3.2 · Algorithm 1 (ReAct-style multimodal retrieval loop)

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During deployment each incoming dialogue turn is summarized into candidate nodes, aligned against the existing store by retrieval, and an LLM decides add / update / no-op for each candidate. Stored node content is therefore rewritten by an LLM at inference time while the decision prompt and procedure remain fixed.

  • Update · Rule-based update

    New nodes are inserted with the fixed Insert rule and the DAG's temporal and pruning constraints are re-applied on every write, so graph content grows under an unchanging update rule (batch size 1, global clustering omitted for latency).

  • Reorganization · Relinking & reindexing

    When a node is updated, ReInsert recomputes its incoming edges and cascades the recomputation to affected children; when a node is removed, orphaned children are re-attached through alternative parents. The organization of the memory graph, not only its content, therefore changes during deployment under a fixed relinking rule.

Related systems