Multimodal Agent MemorySubmit a paper

VoiceMem

VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction

Zhifei Xie, Jiaqi Lang, Ze An, Yifan Zhao, Dongchao Yang, Kai Li, et al. · 2026-08-26

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Audio & speech
Also: text
Representation
Hybrid
The informational left-brain graph, the affective right-brain persona graph, and the optional acoustic nodes are deliberately linked (cross-entity emotion nodes bind to left-brain entities via rho_{v,e}; audio nodes attach to entity nodes) and their retrieval results are merged into a single prompt supplied to the speech model.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. left-brain informational memory: a schema/entity indexing layer over a backend item store (Mem0 by default)

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Hierarchical multi-tier store, Vector-indexed collection
    3. Retrieved byStructure traversal, Similarity search
    4. Source evidenceSource linked & recoverable
    A two-level structure sits above the interchangeable backend item store: schemas give coarse semantic routing and entities locate specific people, events and concepts. Each entity node v carries a textual description d_v, micro-level neighbour links N_v^micro, and an index I_v pointing to the backend memory items it owns. Retrieval traverses schema -> entity -> one-hop neighbours to narrow the candidate set before any backend similarity search, so structure traversal and vector search are composed rather than alternatives, and the entity index recovers the underlying conversational items.Full paper (arXiv HTML): Sec. 3.1 Left Brain · Definition of the left-brain graph G^L; Figure 2 (architecture)

  2. right-brain persona and affective memory: independent persona nodes and cross-entity emotion nodes

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Similarity search
    4. Source evidenceSource linked & recoverable
    Two node types are maintained: independent nodes v^I holding user-intrinsic dispositions and affective tendencies, and cross-entity nodes v_e^C holding context-dependent emotion tied to a left-brain entity through a linking parameter rho_{v,e}. Both node types store natural-language descriptions plus indices into backend memory items, so the persisted affective memory is a text record distilled from the spoken interaction rather than a stored acoustic signal, and it preserves whom or what an emotion concerns.Full paper (arXiv HTML): Sec. 3.2 Right Brain · Equations 4-7

  3. optional multimodal audio nodes attached to entity nodes (speaker voiceprints, acoustic embeddings, raw waveforms)

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal
    4. Source evidenceSource directly available
    Sec. 3.4 states that VoiceMem extends memory beyond text to audio and that, when enabled, agents selectively retain audio as speaker voiceprints, acoustic embeddings, or raw waveforms and attach them as multimodal nodes to the corresponding entity nodes, supporting multi-speaker discrimination, paralinguistic analysis, and environmental sound memory. Sec. 5.3 reports that on the three acoustic categories every text-only system scores between 3.23 and 26.92 while VoiceMem reaches 45.16 to 53.84, which is consistent with non-transcribable acoustic evidence actually being retained and reused.Full paper (arXiv HTML): Sec. 3.4 (audio memory extension); Sec. 5.3 (ChatMem-Bench Paralinguistics & Environment results) · Table 3

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    For each conversational turn x_t an affect estimator produces e_t = phi(x_t); the pair (x_t, e_t) is immediate evidence for adding, editing, or merging persona nodes during the live conversation. Persona memory content therefore changes at inference time under a fixed attribution policy.

  • Consolidation · Merging & deduplication

    Short-horizon updates explicitly include merging persona nodes when new turn-level affective evidence duplicates or subsumes an existing node.

  • Consolidation · Experience abstraction

    After each session, long-horizon attribution jointly analyses the full turn sequence and consolidates recurrent affective evidence into stable independent persona nodes, so transient emotional states are abstracted into durable dispositions instead of accumulating.

  • Revision · LLM-based conflict resolution

    An asynchronous updater extracts new facts from the ongoing conversation and jointly reconciles them against nearby existing memories, choosing among add, update, delete, or keep. Reconciliation of contradictory facts happens during deployment under this fixed decision policy.

  • Forgetting · Discrete eviction

    The same asynchronous reconciliation may resolve to delete, removing superseded memory items from the left-brain store during deployment.

Related systems