Multimodal Agent MemorySubmit a paper

MoMuSE

MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues

Junjie Li, Ke Zhang, Shuai Wang, Kong Aik Lee, Haizhou Li · 2024-12-11

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Audio & speech
Also: Image
Representation
Internalized latent & parametric

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. audio-visual speaker identity state

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    Speaker-identity momentum tracks the target when visual cues disappear.Full-paper method / memory representation and retrieval · PDF p. 6 · Figure 1

How memory changes (T2 operations)

  • Update · Learned update module

    A fixed learned momentum update refreshes speaker state. For MoMuSE, this specifically supports a trained but frozen update module of audio-visual speaker identity state.

Related systems