Multimodal Agent MemorySubmit a paper

Memory-Augmented Object Captioning Agent

Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

Tommaso Galliena, Stefano Rosa, Tommaso Apicella, Pietro Morerio, Alessio Del Bue, Lorenzo Natale · 2026-03-30

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. explored top-down map

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource linked & recoverable
    A top-down explored map grounds current RGB observations in persistent spatial context.Full paper: Model (EPOS-VLM) · PDF p. 4 · Figure 1

  2. object-level episodic tokens

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Serialized object tokens preserve identity and caption history across viewpoints and extended sequences.Full paper: Memory-Augmented Vision–Language Agents · PDF p. 7 · Figure 1

How memory changes (T2 operations)

  • Update · Learned update module

    The autoregressive agent jointly updates object association, captions, and episodic object state. For Memory-Augmented Object Captioning Agent, this specifically supports a trained but frozen update module of explored top-down map.

  • Revision · Rule-based state correction

    Multi-view history is used to maintain persistent identity and correct inconsistent object descriptions. For Memory-Augmented Object Captioning Agent, this specifically supports fixed-rule correction of stored state of explored top-down map.

Related systems