Multimodal Agent MemorySubmit a paper

AUGUSTUS

AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory

Jitesh Jain, Shubham Maheshwari, Ning Yu, Wen-mei Hwu, Humphrey Shi · 2025-10-17

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text, video, audio
Representation
Hybrid
Distinct operative representation paths are deliberately linked or jointly consumed by downstream reasoning, generation, or control.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. semantic tags

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store, Vector-indexed collection
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Encoded language-form descriptions; semantic tags; context nodes with modality, timestamp, URL, and embeddings. CoPe search first retrieves concepts from a contextual tree and personalized tag graph, then retrieves concept-relevant context nodes.Full paper: Storing Concepts in Contextual Memory · PDF p. 6 · Figure 1 / method overview

  2. multimodal context nodes

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inGraph / relational store, Vector-indexed collection
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Encoded language-form descriptions; semantic tags; context nodes with modality, timestamp, URL, and embeddings. CoPe search first retrieves concepts from a contextual tree and personalized tag graph, then retrieves concept-relevant context nodes.Full paper: Storing Concepts in Contextual Memory · PDF p. 6 · Figure 1 / method overview

  3. user context

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Vector-indexed collection
    3. Retrieved bySimilarity search, Structure traversal, Planner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Encoded language-form descriptions; semantic tags; context nodes with modality, timestamp, URL, and embeddings. CoPe search first retrieves concepts from a contextual tree and personalized tag graph, then retrieves concept-relevant context nodes.Full paper: Storing Concepts in Contextual Memory · PDF p. 6 · Figure 1 / method overview

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    Recall memory logs conversations; contextual memory receives new context inserts; in-context memory can be modified. For AUGUSTUS, this specifically supports LLM-mediated memory editing of semantic tags.

  • Forgetting · Discrete eviction

    Recall memory logs conversations; contextual memory receives new context inserts; in-context memory can be modified. For AUGUSTUS, this specifically supports explicit deletion or eviction of stored items of semantic tags.

Related systems