Multimodal Agent MemorySubmit a paper

AdaMM

Beyond Retrieval: Analytic Memory for Multimodal Agents

Zhoujin Tian, Hao Zhang, Yao Tian, Cheng Chen, Yakun Li, Lei Zhang, et al. · 2026-07-31

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Hybrid
The memory-aware joint query planner decomposes a query into analytic operations over the tabular analytic memory and retrieval operations over the hierarchical event memory, routing each to its tool and combining the results for one answer (Sec. 3.5.1-3.5.2, Table 1, Eqs. 9-10).

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. analytic memory: provenance-linked attribute-value observations extracted from dialogue and images, induced into schemas and materialized as queryable tables

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Each interaction round is parsed into attribute-value tuples (e.g. {Sleep Time: 5.5 hrs}) whose provenance pointer p_t refers back to the supporting dialogue span or image; recurring attribute co-occurrences are induced into schemas and materialized as relational tables in which one row corresponds to one interaction round, so the memory is an explicit cross-modal structured store rather than a text blob, and the original image remains reachable through the provenance link.Full paper: 3.3.1 Attribute extraction; 3.3.2 Schema induction; 3.3.3 Memory materialization · arXiv HTML, Sec. 3.3 · Eq. 1; Eqs. 3-7; Table 6 (Appendix E)

  2. retrieval memory: three-tier topic -> episode -> event hierarchy of textual event records carrying interaction order, timestamps and associated image identifiers

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Events store dialogue and contextual content, interaction order, timestamp and associated image identifiers, grouped into episodes and topics. Each node keeps separate textual and visual embeddings, the visual one obtained by mean-pooling the embeddings of associated images (Eq. 12), and node scoring is a late fusion of textual and visual cosine similarities (Eq. 13). The SemanticMatch tool retrieves by similarity, EventLocate retrieves by temporal/order metadata. The visual embeddings serve only as an index into these records, so per the guide they do not open an independent latent representation path.Full paper: 3.4 Retrieval memory; Appendix A · arXiv HTML, Sec. 3.4 and Appendix A · Eq. 12; Eq. 13

How memory changes (T2 operations)

  • Update · Rule-based update

    As new interaction rounds arrive at inference time, candidate pattern mining plus fixed support and all-confidence thresholds decide whether a new schema is created or an existing schema is extended, and the tables are (re)materialized. Persistent memory content and structure therefore change during deployment while the governing thresholds stay fixed (Appendix C states thresholds are fixed across both benchmarks).

  • Reorganization · Relinking & reindexing

    Schema evolution incrementally extends discovered field structures and triggers rematerialization of the corresponding tables, reorganizing how already-stored observations are grouped and addressed for later analytic queries.

Related systems