VisMem
VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
Xinlei Yu, Chengming Xu, Guibin Zhang, Zhangquan Chen, Yudong Zhang, Yongbo He, et al. · 2026-02-05
- Management
- T2 Dynamic Memory
- Memory changes during deployment through a stable management policy.
- Representation
- Internalized latent & parametric
How experience becomes memory
Each path is one memory component, traced from what is stored to how it is found again.
short-term perceptual and long-term semantic latent tokens
- Represented asPersistent multimodal latent stateInternalized latent & parametric
- Stored inHierarchical multi-tier store, Flat sequential records
- Retrieved byLatent / parametric activation
- Source evidenceSource not retained
Special invocation tokens route a context-conditioned query to short- or long-term memory formers, whose generated latent vision tokens are inserted into the autoregressive stream.Full paper: Memory · PDF p. 2
How memory changes (T2 operations)
Update · Learned update module
Trained short- and long-term formers generate new latent memory tokens from visual evidence and current hidden-state queries during inference. For VisMem, this specifically supports a trained but frozen update module of short-term perceptual and long-term semantic latent tokens.