Multimodal Agent MemorySubmit a paper

Stateful Streaming ASR (State Carry-over / State Rollback)

Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents

Sameep Chattopadhyay, Alexander Erdmann, Mari Ostendorf · 2026-08-22

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Audio & speech
Also: text
Representation
Hybrid
The acoustic encoder cache and the decoder/prediction-network state are jointly saved, jointly restored and jointly consumed by the next turn's RNNT decoding step; neither alone constitutes the carried memory.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. cross-utterance acoustic encoder state (attention key-value cache, per-layer convolution cache, valid cache length) of the cache-aware FastConformer-RNNT streaming encoder

    1. Represented asModality-specific encodings & geometric stateSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    The retained inference state includes the streaming encoder's attention key-value cache, a per-layer convolution cache and the corresponding valid cache length. These are audio-derived acoustic activations of the final frames of the previous utterance; they are carried across the turn boundary and re-activated as encoder context for the next utterance instead of being reset. Raw audio of the earlier turn is not retained, so the source signal is not recoverable from the cache.3.1 Model state (with Appendix A.2) · arXiv HTML full text, Section 3.1 · Table 2 (per-component tensor shapes and memory footprint)

  2. cross-utterance decoder state (RNNT prediction-network LSTM hidden/cell state, accumulated decoder hypothesis, last predicted token, number of tokens emitted)

    1. Represented asPersistent multimodal latent stateInternalized latent & parametric
    2. Stored inFlat sequential records
    3. Retrieved byLatent / parametric activation
    4. Source evidenceSource not retained
    A deliberate recurrent memory state is threaded across turns: the LSTM hidden and cell states of the RNNT prediction network together with the accumulated hypothesis, the last predicted token and the emitted-token count are saved at the end of one utterance and restored as the initial decoder state of the next. Algorithm 1 writes S <- S_i after inference on each turn, so the latent state persists across the whole conversational recording rather than within a single utterance.3.1 Model state; 3.3 State-management strategies · arXiv HTML full text, Sections 3.1 and 3.3 · Algorithm 1

How memory changes (T2 operations)

  • Update · Rule-based update

    State Carry-over updates the persistent encoder/decoder memory at every turn boundary by a fixed deterministic rule (S <- S_i after decoding turn i), with no reset. The memory content changes continuously during deployment while the update policy itself is fixed.

  • Forgetting · Discrete eviction

    State Rollback keeps a snapshot of the state at the end of the last substantial turn; when a turn emits fewer than k tokens (k in {1,5,10}, i.e. a backchannel), the accumulated state contributed by that turn is discarded and the saved snapshot is restored. This is an inference-time, fixed-threshold eviction of memory content that would otherwise persist.

Related systems