fix(interruption): keep an open overlap when a second agent speech segment starts - #6550
fix(interruption): keep an open overlap when a second agent speech segment starts#6550biztex wants to merge 1 commit into
Conversation
…gment starts One agent turn can span several speech segments: a reply that produces tool calls skips _on_end_of_agent_speech, so the follow-up tool-response reply (or a queued say()) raises _AgentSpeechStartedSentinel again with no end sentinel in between. _forward_data treated every start sentinel as a new turn and reset the stream state, clearing _overlap_started while the user was still mid-interrupt. From that point the interruption was silently swallowed: _overlap_started gates audio forwarding to the bargein gateway, and only a fresh VAD speech onset - which never comes for speech already under way - can raise it again, so no verdict was ever emitted and the agent talked straight through the user. Skip the reset when the stream already believes the agent is speaking and an overlap is open, treating the sentinel as a continuation of the same turn. A genuine new turn (an end sentinel was seen) still wipes the overlap, the cache and the counters. Ports the stream-side guard from livekit/agents-js#2117 (the transport-failover half of that PR does not apply here: these flags live on self and _run() reconnects in place on the same stream object). Fixes livekit#6548
ba01151 to
510f401
Compare
|
TL;DR for reviewers One agent turn can span several speech segments (the reply after a tool call, or a queued The change is a narrow continuation guard in Status: all checks green, CLA signed, no open review threads. Happy to adjust anything. |
|
Superseded by #6564, which fixes the root cause at the signal source (real speech boundaries around the thinking gap, restarting overlap inference on resume) and also covers the backchannel-metrics and remote-session angles. Closing in its favor. |
Fixes #6548
Problem
One agent turn can span several speech segments: a reply that produces tool calls deliberately skips
_on_end_of_agent_speech, so the follow-up tool-response reply (or a queuedsay()) raises_AgentSpeechStartedSentinela second time with no end sentinel in between._forward_datatreated every start sentinel as a brand-new turn and called_reset_state(), clearing_overlap_startedwhile the user was mid-interruption.From that moment the interruption is silently swallowed:
_overlap_startedgates whether user audio is forwarded to the bargein gateway at all, so classification stops;_OverlapSpeechStartedSentinel, which only comes from a fresh VAD speech onset — and VAD does not re-announce speech that is already under way;_OverlapSpeechEndedSentinelthe event emission is behindif self._overlap_started, so noOverlappingSpeechEventis emitted andAgentActivity.on_interruptionnever runs. The agent talks straight through the user.Fix
Skip the reset when the stream already believes the agent is speaking and an overlap is open — i.e. treat the sentinel as a continuation of the same turn. The guard is deliberately narrow: a genuine new turn (an end sentinel was seen, so
_agent_speech_startedisFalse) still wipes the overlap, the audio buffer, the cache and the counters.This ports the stream-side guard from the JS fix (livekit/agents-js#2117). The other half of that PR — restoring overlap state on a transport-failover replacement stream — does not apply to Python: these flags live on
selfand_run()reconnects in place on the same stream object.Tests
New
tests/test_interruption/test_interruption_overlap_state.py(unit, hermetic — fake gateway WS that answers every batch), ported from the JS PR's regression suite:test_second_speech_segment_keeps_open_overlap— user audio keeps reaching the gateway after a mid-overlap second segment, and the end-of-overlap event still surfaces withnum_requests > 0(fails onmain)test_bargein_verdict_after_second_speech_segment— abargein_detectedresponse after the second segment still producesis_interruption=True(fails onmain)test_new_agent_turn_still_resets_overlap_state— a genuine new turn still resets: audio stays gated until a fresh overlap opens (passes onmainand with the fix — pins the guard's narrowness)Verification: full
--unitsuite passes (1274 passed),ruff format --check/ruff checkclean,mypystrict clean.🤖 Generated with Claude Code