Multimodal Agent MemorySubmit a paper

KARMA

KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems

Zixuan Wang, Bo Yu, Junzhe Zhao, Wenhao Sun, Sai Hou, Shuai Liang, et al. · 2024-09-23

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Hybrid
Distinct operative representation paths are explicitly linked for downstream retrieval, reasoning, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. 3D scene graph experience

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource linked & recoverable
    Long-term memory holds comprehensive 3D scene graphs.Full paper: Method · PDF p. 3 · Figure 1

  2. dynamic object positions and states

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    Short-term memory records changing object states.Full paper: Method · PDF p. 3 · Figure 1

How memory changes (T2 operations)

  • Update · Rule-based update

    A fixed rule updates short-term state. For KARMA, this specifically supports fixed-rule insertion or replacement of 3D scene graph experience.

  • Forgetting · Discrete eviction

    Adaptive replacement discards less pertinent entries. For KARMA, this specifically supports explicit deletion or eviction of stored items of 3D scene graph experience.

Related systems