Multimodal Agent MemorySubmit a paper

GEMS

GEMS: Agent-Native Multimodal Generation with Memory and Skills

Zefeng He, Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Yu Cheng, et al. · 2026-03-30

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. trajectory factual state

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Agent Memory persistently stores factual states across the iterative generation trajectory.Full-paper method / memory representation and retrieval · PDF p. 5 · Figure 1

  2. compressed experiential summaries

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Hierarchically compressed summaries preserve a global view of the optimization process while reducing redundancy.Full-paper method / memory representation and retrieval · PDF p. 2 · Figure 1

How memory changes (T2 operations)

  • Consolidation · Episodic / hierarchical summarization

    Trajectory experience is compressed into hierarchical summaries. For GEMS, this specifically supports episodic or hierarchical summarization of trajectory factual state.

  • Consolidation · Experience abstraction

    Repeated generation traces are distilled into reusable experiential memory. For GEMS, this specifically supports abstraction of episodes into reusable experience of trajectory factual state.

Related systems