Multimodal Agent MemorySubmit a paper

VIGA

Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning

Shaofeng Yin, Jiaxin Ge, Zora Zhiruo Wang, Xiuyu Li, Michael J. Black, Trevor Darrell, et al. · 2026-01-16

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. render and comparison history

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    The write-run-render-compare loop retains rendered evidence across iterative inverse-graphics reasoning.Full paper: Memory · PDF p. 3 · Figure 1

  2. plans, programs, and code-diff memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Plans, executable programs, diffs, and render history form an evolving context and skill library.Full paper: Memory · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, VIGA applies fixed-rule insertion or replacement to render and comparison history; the memory state changes while the governing update policy remains fixed.

  • Update · Prompted LLM/VLM record update

    Execution feedback triggers fixed-loop revision of plans and code. For VIGA, this specifically supports LLM-mediated memory editing of render and comparison history.

  • Forgetting · Discrete eviction

    During operation, VIGA applies explicit deletion or eviction of stored items to render and comparison history; the memory state changes while the governing update policy remains fixed.

Related systems