Multimodal Agent MemorySubmit a paper

ViLoMem

ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory

Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu, Qunyi Xie, Xiao Tan, et al. · 2025-11-26

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. dual-stream semantic memory

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved bySimilarity search, Structure traversal
    4. Source evidenceSource linked & recoverable
    Visual-distraction and logical-reasoning error schemas are grown and refined from successes and failures.Full paper: 2.2. Long-term Memory · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    During operation, ViLoMem applies LLM-mediated memory editing to dual-stream semantic memory; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Merging & deduplication

    During operation, ViLoMem applies merging or deduplication of overlapping entries to dual-stream semantic memory; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Experience abstraction

    Task outcomes are abstracted into reusable semantic strategies and error schemas. For ViLoMem, this specifically supports abstraction of episodes into reusable experience of dual-stream semantic memory.

  • Revision · LLM-based conflict resolution

    New evidence refines previously stored semantic knowledge. For ViLoMem, this specifically supports LLM-based resolution of conflicting records of dual-stream semantic memory.

Related systems