voice: support localized approvals and reliable GPT Live handoffs #343301

Fixes #343299, #342526.

Changes

• Recognize complete approval/rejection phrases in the selected voice language across all 25 supported speech-recognition languages, including Automatic-mode languages and simplified/traditional Chinese variants. English commands remain available as a fallback.


• Keep parsing deterministic: no language-model classification, substring matching, or inferred consent. Qualified/ambiguous replies remain unresolved; existing stale-prompt and duplicate-response safeguards are preserved.


• Accept whole repetitions of the same explicit approval/rejection phrase when speech recognition combines retries into one transcript, such as “Approved Approved.” Mixed, qualified, and concatenated commands remain unresolved.


• Preserve successful new-session sends when navigating to the running session disposes the composer input. Skip post-send draft persistence for the disposed input instead of incorrectly reporting that the request failed.


• Normalize displayed question choices using the active voice locale, including Turkish dotted/dotless I, while preserving English ordinal commands. Omit response guidance until a real voice locale is known.


• Read pending confirmations with localized response guidance. For question forms, read the prompt and numbered choices and invite a voice reply or use of the controls in chat, including follow-up questions. Do not require English for question replies.


• Use GPT Live’s documented Generate <|SpawnThinking|> handoff cue at startup and in runtime instructions, including file-creation requests from an empty new-session composer. Remove conflicting instructions to handle session navigation without the client, and require the client result before acknowledging acceptance.


• Apply GPT Live language changes immediately without reconnecting.


• Add coverage for every supported recognition language, Unicode normalization, Turkish casing, ambiguous/negated replies, pending-prompt identity, localized question forms, and the German “verwerfen” regression without premature acknowledgement.

GPT Live handoff

The reported failure had a complete transcript but no client delegation: GPT Live spoke an acknowledgement without starting the coding session. Both prompts now follow OpenAI’s documented delegation policy, using Generate <|SpawnThinking|> instead of relying on the phrase “delegate to the client.” Startup instructions now agree that the client owns session routing. Conversational speech is not automatically submitted, and deterministic approval and stale-prompt safeguards are unchanged. This remains model-driven delegation, not a protocol-level guarantee.

Question reply language

When the agent supplies a question and choices in the user's language, the user can answer by voice in that language using an exact displayed choice label, a choice number, or free text where the question permits it. For example, a German question with “Westen” and “Osten” accepts “Osten”; a following text question accepts “Meine Bereitstellung.” English replies remain available, but are not required.

This is not unrestricted multilingual natural-language parsing. Shorthand commands such as “skip,” “other,” and “second,” and the multi-choice conjunction “and,” remain English-only. Localized paraphrases or ordinal words are not added by this change.

The question text and choices come from the chat agent. Changing the VS Code locale or agents.voice.language does not itself translate that content; localized replies work when the supplied choices are localized, and free-text answers preserve the user's language.

Validation

npm run build-fast -- --client-only

Result: passed.

./scripts/test.sh --reporter dot \

  --run src/vs/sessions/contrib/chat/test/browser/sharedChatInputComposer.test.ts \

  --run src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceClientService.test.ts \

  --run src/vs/workbench/contrib/chat/test/common/voiceClient/voiceLiveInputRouter.test.ts \

  --run src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceToolDispatchService.test.ts \

  --run src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceSessionController.test.ts

Result: 375 passing.

npm run typecheck-client

Result: passed.

npm run eslint -- \

  src/vs/workbench/contrib/chat/browser/voiceClient/voiceClientService.ts \

  src/vs/workbench/contrib/chat/common/voiceClient/voiceLiveInputRouter.ts \

  src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceClientService.test.ts \

  src/vs/workbench/contrib/chat/test/common/voiceClient/voiceLiveInputRouter.test.ts

Result: passed.

node build/hygiene.ts \

  src/vs/sessions/contrib/chat/browser/sharedChatInputComposer.ts \

  src/vs/sessions/contrib/chat/test/browser/sharedChatInputComposer.test.ts \

  src/vs/workbench/contrib/chat/browser/voiceClient/voiceClientService.ts \

  src/vs/workbench/contrib/chat/common/voiceClient/voiceLiveInputRouter.ts \

  src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceClientService.test.ts \

  src/vs/workbench/contrib/chat/test/common/voiceClient/voiceLiveInputRouter.test.ts

Result: passed; all six changed files checked, including intentional Unicode phrases.

Latest handoff validation

npm run build-fast

Result: passed; refreshed client and Copilot extension output.

cd extensions/copilot

npm run test:unit -- src/extension/byok/vscode-node/test/gptLiveSession.spec.ts

npx eslint src/extension/byok/vscode-node/gptLiveSession.ts src/extension/byok/vscode-node/test/gptLiveSession.spec.ts

Result: 13 tests passed; extension ESLint passed.

./scripts/test.sh --reporter dot \

  --run src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceClientService.test.ts \

  --run src/vs/workbench/contrib/chat/test/common/voiceClient/voiceLiveInputRouter.test.ts \

  --run src/vs/sessions/contrib/chat/test/browser/sharedChatInputComposer.test.ts

Result: 159 passing after the handoff prompt changes.

node build/hygiene.ts src/vs/workbench/contrib/chat/browser/voiceClient/voiceClientService.ts src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceClientService.test.ts extensions/copilot/src/extension/byok/vscode-node/gptLiveSession.ts extensions/copilot/src/extension/byok/vscode-node/test/gptLiveSession.spec.ts

Result: passed for the two core files; root hygiene excludes Copilot extension files, which were linted separately above.

Live OSS verification: reloaded the rebuilt renderer and extension host, connected to real GPT Live, and supplied locally synthesized speech requesting a new file while requiring a question before creation. Observed session.delegation.created, send_to_chat with the recognized request, successful new-composer dispatch, and a newly opened coding session asking for file contents. No file creation was authorized. The original microphone track was restored and Voice Mode was disconnected after the check.

Manual verification

1. Set agents.voice.language to de with an English display language and connect GPT Live Voice Mode.


2. Trigger a tool confirmation, say “verwerfen,” and verify the actual pending action is rejected before the voice agent acknowledges success. Repeat with a new confirmation and “ja” to approve it.


3. Say a qualified reply such as “ja, aber warte” and verify no action is applied and the user is prompted to reply explicitly or use the controls.


4. Trigger a German question form with localized choices and a following text question. Verify the prompt and numbered choices are read without an English-only restriction, answer with an exact German choice label or its number, and answer the text question in German. Check that follow-up questions also invite a voice reply or use of the controls in chat.


5. Change the voice language while connected, and repeat using Automatic mode with a non-English display language.


6. Start a session from the new-session picker and verify successful dispatch is not reported as a failure when the composer is disposed. Barge into confirmation narration and repeat the same explicit approval; if both attempts arrive as one transcript, verify the confirmation resolves once. Mixed or qualified responses must not approve it.

Regression tests reproduce the observed composer-disposal error and combined “Approved Approved” transcript before the fixes. The new-session handoff was verified against real GPT Live using synthesized audio as described above. Full live microphone verification of multilingual approvals and native-speaker review of the phrase translations were not performed. Automated tests verify exact matching and routing, not linguistic completeness.