Multimodal Agent MemorySubmit a paper

ROVER

ROVER: Routing Object-Centric Visual Evidence for Grounded Multi-Image Reasoning

Guannan Lv, Ren Nie, Hongjian Dou, Tingting Gao · 2026-05-28

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Visual Working Space (VWS): an append-ordered store of object-anchored visual features distilled by the Sift module, shared across all routing events in a reasoning trajectory

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceRetained but not accessible to the agent
    Whenever the model emits a grounding event, Sift applies object-centric differential attention over non-RoI patch tokens to distill an object-anchored feature t_k^Sift, which is concatenated into the Visual Working Space W_k = [W_{k-1}; t_k^Sift] in R^{k x d}, with W_0 = empty. The VWS is shared across all routing events irrespective of source image index, forming a global memory of grounded objects for the whole multi-image trajectory. It is read by the Weave module, t_k^Weave = Enc_Weave(t_k^Sift, W_k, M_k^Weave), which cross-attends from the current object feature over the accumulated history under an autoregressive mask and injects the result back into the reasoning stream. Entries are compressed hidden vectors that do not themselves carry box coordinates or image indices (those live in the emitted text tokens), so the source image region is not recoverable from a stored entry.Full paper (arXiv HTML v2): Sec. 3.1 Sift and Weave modules; Eq. 5 (VWS update) and Eq. 6 (Weave read) · HTML Sec. 3.1 · Equations 5 and 6

How memory changes (T2 operations)

  • Update · Rule-based update

    During inference the VWS changes by a fixed append rule: each new grounding event concatenates its Sift feature onto the existing store, which is then re-read by Weave at every subsequent routing event. Memory content therefore evolves across inference steps while the write rule (append) and the read rule (masked cross-attention) stay fixed. No eviction, merging, or reweighting is performed.

Related systems