Multimodal Agent MemorySubmit a paper

Memorize-and-Generate

Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation

Tianrui Zhu, Shiyi Zhang, Zhirui Sun, Jingqi Tian, Yansong Tang · 2025-12-23

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Video & streaming
Also: text
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Structural anchors (final frame of each attention block) for long-term consistency

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceRetained but not accessible to the agent
    Autoencoded KV-cache entries; Anchor frames. Ongoing generation is conditioned on the cached anchors (read internally during decoding).Full-paper method / memory representation and retrieval · PDF p. 8 · Figure 1 / method overview

  2. Structural anchors (final frame of each attention block) for long-term consistency

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Autoencoded KV-cache entries; Anchor frames. Ongoing generation is conditioned on the cached anchors (read internally during decoding).Full-paper method / memory representation and retrieval · PDF p. 8 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Learned update module

    During operation, Memorize-and-Generate applies a trained but frozen update module to Structural anchors (final frame of each attention block) for long-term consistency; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Merging & deduplication

    During operation, Memorize-and-Generate applies merging or deduplication of overlapping entries to Structural anchors (final frame of each attention block) for long-term consistency; the memory state changes while the governing update policy remains fixed.

  • Forgetting · Discrete eviction

    KV-cache entries are maintained, compressed, and evicted by a fixed learned mechanism as the stream proceeds. For Memorize-and-Generate, this specifically supports explicit deletion or eviction of stored items of Structural anchors (final frame of each attention block) for long-term consistency.

Related systems