Multimodal Agent MemorySubmit a paper

SAVEMem

Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding

Hang Wu, Sherin Mary Mathews, Yujun Cai, Ming-Hsuan Yang, Yiwei Wang · 2026-05-08

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Three-tier visual token memory (M_short anchor, M_mid, M_long) holding original frame hidden states together with cached semantic salience scores under a fixed global token budget

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved bySimilarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceRetained but not accessible to the agent
    Memory is a three-tier store: a 4-frame short-term anchor at full fidelity, a 16-frame mid-term tier compressed by temporal pruning, and a long-term tier holding spatially selective tokens, all inside a global budget of B = 2048 tokens independent of video length. Retained tokens carry their original hidden states with no synthetic averaging, so what persists is the visual encoding rather than the frame pixels, which are not recoverable. Each token is indexed by a salience score s(v) = max_{q in Q} cos(v, q) against a fixed pseudo-question bank, computed once at encoding and cached. Retrieval ranks frames by ColBERT-style late interaction sigma(g) = (1/|g|) sum_i max_j cos(g_i, q_j) against the live query, and an anchor-conditioned recency gate can short-circuit to the short-term tier when MaxSim(M_short, q) >= rho * EMA of recent short-term scores, making tier recency an operative retrieval signal alongside similarity.Full paper (arXiv HTML v1): Sec. 3.2 memory hierarchy and scoring; Eq. 3 (late interaction); Algorithm 1 · HTML Sec. 3.2 · Algorithm 1; Equation 3

How memory changes (T2 operations)

  • Update · Rule-based update

    For every arriving frame f_t the system encodes V_t, computes s_t = MaxSim(V_t, Q) and appends the scored frame to the short-term tier, so memory content changes continuously during streaming under a fixed write rule.

  • Consolidation · Merging & deduplication

    When the short-term tier overflows, the oldest frame is compressed on its way out by removing tokens that are already well represented by their temporal neighbours while sparing high-salience tokens, i.e. redundant duplicated visual content is collapsed at streaming time under a fixed criterion.

  • Reorganization · Memory-level migration (short- to long-term)

    Frames migrate short -> mid on short-term overflow and mid -> long on mid-term overflow, being progressively compressed (temporal pruning, then spatially dispersed selection ranked by s(v)) as they descend, so the organization of memory across tiers changes during streaming under a fixed FIFO-triggered rule.

  • Forgetting · Discrete eviction

    While the total token count over the three tiers exceeds the budget B, the lowest-salience token in the long-term tier is discarded, permanently removing items from memory during deployment under a deterministic rule (no decay function is used).

Related systems