Multimodal Agent MemorySubmit a paper

GRGA (Graph-based Retrieval-Generation Agent)

Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding

Quanwei Tang, Dong Zhang, Shoushan Li, Guodong Zhou · 2026-07-02

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Audio & speech
Also: text
Representation
Hybrid
The pre-built utterance graph, the timestamp-addressed raw waveform, and the accumulated textual belief/evidence state are deliberately linked (graph nodes carry the timestamps used to fetch audio, and both feed the same evidence set) and jointly supplied to the Answer Synthesizer.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. multi-dimensional heterogeneous meeting graph over utterance nodes (transcript, speaker identity, start/end timestamps, LLM-distilled speaker profile) with temporal, reply-to, same-speaker, entity and coreference-semantic edges

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store
    3. Retrieved byStructure traversal, Planner / tool-assisted retrieval, Similarity search, Metadata / spatiotemporal retrieval
    4. Source evidenceSource linked & recoverable
    VAD + ASR + forced alignment turn the meeting audio into timestamped utterance nodes; ERes2NetV2 speaker embeddings and incremental clustering assign speaker IDs, and an LLM distills a structured speaker profile stored as a node attribute. The heterogeneous graph G=(V,E) adds temporal, reply-to, same-speaker, entity co-occurrence and LLM-detected coreference edges, and this graph is the environment the agent queries. Nodes carry timestamps and utterance IDs, so the underlying audio span is recoverable.3.2 Audio-to-Graph Indexing; 3.2.2 Multi-dimensional Graph Modeling · PDF pp. 20718-20719 · Figure 3 (framework overview, bottom-left graph)
    The planner issues keyword_search (BM25), semantic_search (dense retrieval), hybrid_search, filte_time_range, filte_speaker and traverse_relations over the graph, i.e. tool-mediated similarity, metadata and edge-traversal access rather than a single one-shot retrieval.3.3 GRGA Planning Process; Appendix (Tools Details) · PDF pp. 20719, 20734 · Table 11 (atomic tools in the action space)

  2. retained raw meeting waveform, addressed by timestamp through the audio_segment tool

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval, Planner / tool-assisted retrieval
    4. Source evidenceSource directly available
    audio_segment(t_start, t_end) 'retrieves raw waveform data to ground text in acoustic signals (e.g., for emotion detection)'. The tool-usage analysis states that acoustic-aware questions show a high frequency of Audio Access calls in the final steps, i.e. the agent listens to the retained raw audio when transcripts are insufficient; removing the tool causes the second-largest accuracy drop.Appendix (Tools Details); 4 Experimentation (ablation and tool-usage analysis) · PDF pp. 20734, 20720-20721 · Table 11 (audio_segment tool); Table 5 (ablation: w/o Tool: Audio Access, -14.78 accuracy on AMI)

  3. agent belief state b_t = {query, decomposed constraints, interaction history H_t, accumulated evidence set, reflector critiques}

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    The task is cast as a POMDP whose belief state b_t, explicitly defined as the sufficient statistic for decision-making, is initialized from the decomposed query and updated by a deterministic transition b_{t+1} = psi(b_t, P_t, o_t) = b_t + {P_t, o_t}. Observations are textual evidence segments s_i = (text_i, time_i, spk_i) retained across planning iterations and consumed by the Answer Synthesizer with citations; utterance IDs and timestamps link each record back to the graph and audio.3.1 Problem Formulation; 3.3 GRGA Planning Process · PDF pp. 20718-20719 · Equations 2, 5, 6, 7

How memory changes (T2 operations)

  • Update · Rule-based update

    At every inference step the retained belief/evidence state is updated by a fixed deterministic transition function psi that appends the executed plan and the newly retrieved evidence; the memory content grows across steps while the update rule itself is fixed and training-free.

  • Revision · Evidence-weighted belief revision

    The Self-Reflector scores the entailment between accumulated evidence and the candidate answer; when the score falls below the threshold the negative critique is injected into the belief state (b_{t+1} <- b_{t+1} union {fb_t}), which revises the retained evidence/belief content and forces a corrective plan on the next iteration. The revision policy is a fixed prompted rule, not a learned one.

Related systems