Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory
Authors
* Equal first authors§ Project organizers† Corresponding author‡ Core authorsContact: minghao.guo@rutgers.edu, qjiao@nd.edu, zyye@fudan.edu.cn
37 institutions
- Rutgers University
- University of Notre Dame
- University of Michigan
- Fudan University
- MBZUAI
- JHU
- Peer AI
- UCSD
- Georgia Institute of Technology
- SemiAnalysis
- University of Southern California
- LMSYS Org
- NUS
- Capital One
- Independent Researcher
- CMU
- UIC
- KAIST
- Zoom
- NVIDIA
- UPenn
- MSU
- Duke
- UTD
- Cornell
- Modelslive Inc.
- Princeton
- University of Central Florida
- Squirrel AI
- NTU
- Purdue
- TikTok
- Dartmouth College
- UIUC
- University of Washington
- Humani Labs
- Meta
Abstract
Multimodal models and agents have emerged to enable AI systems to learn from and interact with the real world more like humans by integrating information across multiple modalities. However, enabling multimodal agents to continually acquire useful information from complex, heterogeneous multimodal data remains a significant challenge as they move toward long-horizon tasks and lifelong settings. Addressing this challenge requires effective memory mechanisms that can maintain continuity across sessions, adapt to changing users and environments, and refine memory-management policies through accumulated experience.
We introduce a two-axis analytical framework for multimodal agent memory structured around two key questions: how multimodal information is formed and represented (Axis I, Formation and Representation), and how its management adapts with accumulated experience (Axis II, Memory Management). Axis I characterizes what information is retained and how it is encoded (e.g., modality-specific embeddings, textual and structured records, latent and parametric representations) to support subsequent reasoning and action. Axis II distinguishes three tiers: T1 Static Memory reuses fixed memory; T2 Dynamic Memory updates memory through a stable policy; and T3 Self-Evolving Memory updates memory through an evolving policy.
Based on this framework, we synthesize recurring design patterns and derive principles for balancing source fidelity with abstraction, maintaining temporal validity and provenance, grounding memory use in recoverable evidence, and evolving memory policies over time. Furthermore, we discuss the challenges and potential future directions for the memory mechanism of multimodal agents.
Cite
@article{guo2026seeing,
title = {Seeing, Maintaining, and Learning: {The Evolution of Multimodal Agent Memory}},
author = {Guo, Minghao and Jiao, Qingyue and Liu, Xiujin and Pang, Jianhong and Yang, Ruimeng and Xu, Naihao and Che, Liwei and Xue, Haochen and Wang, Ziqi and Cai, Xinying and Luo, Jiaxuan and Deng, Jiaxin and Ye, Xuesong and Qu, Jintao and Gao, Wenyao and Gu, Jingwen and Zhao, Chenyang and Liu, Shichun and Zhang, Guibin and Liu, Zefang and Wang, Siyin and Xu, Wujiang and Li, Danrui and Shi, Zeru and Yeo, Woongyeong and Liu, Shuang and Huang, Wei-Chieh Jason and Zhang, Weizhi and Yang, Tianyu and Kim, Kangsan and Hu, Yebowen and Wen, Bowen and Jiang, Bowen and Su, Haohao and Yin, Ming and Yu, Sheldon and Zhang, Mian and Chen, Yuyan and Zhang, Hanrong and Mei, Kai and Zhang, Zihao and Jin, Mingyu and Zhu, Xi and Wang, Hengyi and Wei, Tianxin and Nan, Linyong and Liang, Zhenwen and Chu, Zhendong and Huang, Ziqi and Su, Yongye and Dong, Sixun and Liu, Shujian and Liang, Kaiqu and Han, Jiaojiao and Fei, Zhaoye and Wang, Kun and Dou, Shihan and Guo, Zhichun and Diao, Xingjian and Gu, Zeqi and Wang, Heng and Feng, Tao and Cheng, Qian and Palangi, Hamid and Xu, Silei and Yun, Xi and Zhu, Yi and Liu, Shilong and Wen, Qingsong and Hwang, Sung Ju and Wu, Zuxuan and Jiang, Yu-Gang and Shi, Yiyu and Metaxas, Dimitris N. and You, Jiaxuan and McAuley, Julian and Wang, Mengdi and Han, Jiawei and Yu, Philip S. and Ye, Ziyi},
journal = {arXiv preprint arXiv:2610.19366},
year = {2026},
eprint = {2610.19366},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2610.19366}
}