Multimodal Agent MemorySubmit a paper

Dynam3D

Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation

Zihan Wang, Seungjun Lee, Gim Hee Lee · 2025-01-01

Read the paper

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: text, Image, video
Representation
Source- and modality-specific

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Patch-level 3D feature points

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byStructure traversal
    4. Source evidenceRetained but not accessible to the agent
    Projects 2D features into 3D space.; Merges 2D instance masks into geometry-consistent 3D instances.; Aggregates instance information into zone-level tokens.; Updates feature points with frustum culling to refresh outdated views.. graph query/traversal over the memory conditioned on the current goal.Full-paper method / memory representation and retrieval · PDF p. 5 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Rule-based update

    During operation, Dynam3D applies fixed-rule insertion or replacement to Patch-level 3D feature points; the memory state changes while the governing update policy remains fixed.

  • Consolidation · Merging & deduplication

    During operation, Dynam3D applies merging or deduplication of overlapping entries to Patch-level 3D feature points; the memory state changes while the governing update policy remains fixed.

  • Forgetting · Discrete eviction

    During operation, Dynam3D applies explicit deletion or eviction of stored items to Patch-level 3D feature points; the memory state changes while the governing update policy remains fixed.

Related systems