2:I[7012,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],"MarkdownRenderer"] 4:I[9856,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],""] 5:I[4126,[],""] 7:I[9630,[],""] 8:I[4278,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"HeadingProvider"] 9:I[1476,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Header"] a:I[3167,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Sidebar"] b:I[7409,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"PageFrame"] 3:T4078, # Thinker-Talker Frontend Hooks > **Location:** `apps/web-app/src/hooks/` > **Status:** Production Ready > **Last Updated:** 2025-12-01 ## Overview The Thinker-Talker frontend integration consists of several React hooks that manage WebSocket connections, audio capture, and playback. These hooks provide a complete voice mode implementation. ## Hook Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Voice Mode Components │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ useThinkerTalkerVoiceMode │ │ │ │ (High-level orchestration hook) │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────┴───────────────┐ │ │ ▼ ▼ │ │ ┌─────────────────────────┐ ┌─────────────────────────┐ │ │ │ useThinkerTalkerSession │ │ useTTAudioPlayback │ │ │ │ (WebSocket + Protocol) │ │ (Audio Queue + Play) │ │ │ └─────────────────────────┘ └─────────────────────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌─────────────────────────┐ ┌─────────────────────────┐ │ │ │ WebSocket API │ │ Web Audio API │ │ │ │ (Backend T/T) │ │ (AudioContext) │ │ │ └─────────────────────────┘ └─────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ``` ## useThinkerTalkerSession Main hook for WebSocket communication with the T/T pipeline. ### Import ```typescript import { useThinkerTalkerSession } from "../hooks/useThinkerTalkerSession"; ``` ### Usage ```typescript const { status, error, transcript, partialTranscript, pipelineState, currentToolCalls, metrics, connect, disconnect, sendAudioChunk, bargeIn, } = useThinkerTalkerSession({ conversation_id: "conv-123", voiceSettings: { voice_id: "TxGEqnHWrfWFTfGW9XjX", language: "en", barge_in_enabled: true, }, onTranscript: (t) => console.log("Transcript:", t.text), onResponseDelta: (delta, id) => appendToChat(delta), onAudioChunk: (audio) => playAudio(audio), onToolCall: (tool) => showToolUI(tool), }); ``` ### Options ```typescript interface UseThinkerTalkerSessionOptions { conversation_id?: string; voiceSettings?: TTVoiceSettings; onTranscript?: (transcript: TTTranscript) => void; onResponseDelta?: (delta: string, messageId: string) => void; onResponseComplete?: (content: string, messageId: string) => void; onAudioChunk?: (audioBase64: string) => void; onToolCall?: (toolCall: TTToolCall) => void; onToolResult?: (toolCall: TTToolCall) => void; onError?: (error: Error) => void; onConnectionChange?: (status: TTConnectionStatus) => void; onPipelineStateChange?: (state: PipelineState) => void; onMetricsUpdate?: (metrics: TTVoiceMetrics) => void; onSpeechStarted?: () => void; onStopPlayback?: () => void; autoConnect?: boolean; } ``` ### Return Values | Field | Type | Description | | ------------------- | ----------------------------- | ---------------------- | | `status` | `TTConnectionStatus` | Connection state | | `error` | `Error \| null` | Last error | | `transcript` | `string` | Final user transcript | | `partialTranscript` | `string` | Streaming transcript | | `pipelineState` | `PipelineState` | Backend pipeline state | | `currentToolCalls` | `TTToolCall[]` | Active tool calls | | `metrics` | `TTVoiceMetrics` | Performance metrics | | `connect` | `() => Promise` | Start session | | `disconnect` | `() => void` | End session | | `sendAudioChunk` | `(data: ArrayBuffer) => void` | Send audio | | `bargeIn` | `() => void` | Interrupt AI | ### Types ```typescript type TTConnectionStatus = | "disconnected" | "connecting" | "connected" | "ready" | "reconnecting" | "error" | "failed" | "mic_permission_denied"; type PipelineState = "idle" | "listening" | "processing" | "speaking" | "cancelled"; interface TTTranscript { text: string; is_final: boolean; timestamp: number; message_id?: string; } interface TTToolCall { id: string; name: string; arguments: Record; status: "pending" | "running" | "completed" | "failed"; result?: unknown; } interface TTVoiceMetrics { connectionTimeMs: number | null; sttLatencyMs: number | null; llmFirstTokenMs: number | null; ttsFirstAudioMs: number | null; totalLatencyMs: number | null; sessionDurationMs: number | null; userUtteranceCount: number; aiResponseCount: number; toolCallCount: number; bargeInCount: number; reconnectCount: number; sessionStartedAt: number | null; } interface TTVoiceSettings { voice_id?: string; language?: string; barge_in_enabled?: boolean; tts_model?: string; } ``` ### Reconnection The hook implements automatic reconnection with exponential backoff: ```typescript const MAX_RECONNECT_ATTEMPTS = 5; const BASE_RECONNECT_DELAY = 300; // 300ms const MAX_RECONNECT_DELAY = 30000; // 30s // Delay calculation delay = min((BASE_DELAY * 2) ^ attempt, MAX_DELAY); ``` Fatal errors (mic permission denied) do not trigger reconnection. ## useTTAudioPlayback Handles streaming PCM audio playback with queue management. ### Import ```typescript import { useTTAudioPlayback } from "../hooks/useTTAudioPlayback"; ``` ### Usage ```typescript const { isPlaying, queuedChunks, currentLatency, playAudioChunk, stopPlayback, clearQueue, getAudioContext } = useTTAudioPlayback({ sampleRate: 24000, onPlaybackStart: () => console.log("Started playing"), onPlaybackEnd: () => console.log("Finished playing"), onError: (err) => console.error("Playback error:", err), }); // Queue audio from WebSocket function handleAudioChunk(base64Audio: string) { const pcmData = base64ToArrayBuffer(base64Audio); playAudioChunk(pcmData); } // Handle barge-in function handleBargeIn() { stopPlayback(); clearQueue(); } ``` ### Options ```typescript interface UseTTAudioPlaybackOptions { sampleRate?: number; // Default: 24000 bufferSize?: number; // Default: 4096 onPlaybackStart?: () => void; onPlaybackEnd?: () => void; onError?: (error: Error) => void; } ``` ### Return Values | Field | Type | Description | | ----------------- | ----------------------------- | ----------------------- | | `isPlaying` | `boolean` | Audio currently playing | | `queuedChunks` | `number` | Chunks waiting to play | | `currentLatency` | `number` | Playback latency (ms) | | `playAudioChunk` | `(data: ArrayBuffer) => void` | Queue chunk | | `stopPlayback` | `() => void` | Stop immediately | | `clearQueue` | `() => void` | Clear pending chunks | | `getAudioContext` | `() => AudioContext` | Get context | ### Audio Format Expects 24kHz mono PCM16 (little-endian): ```typescript // Convert base64 to playable audio function base64ToFloat32(base64: string): Float32Array { const binary = atob(base64); const bytes = new Uint8Array(binary.length); for (let i = 0; i < binary.length; i++) { bytes[i] = binary.charCodeAt(i); } // Convert PCM16 to Float32 for Web Audio const pcm16 = new Int16Array(bytes.buffer); const float32 = new Float32Array(pcm16.length); for (let i = 0; i < pcm16.length; i++) { float32[i] = pcm16[i] / 32768; } return float32; } ``` ## useThinkerTalkerVoiceMode High-level orchestration combining session and playback. ### Import ```typescript import { useThinkerTalkerVoiceMode } from "../hooks/useThinkerTalkerVoiceMode"; ``` ### Usage ```typescript const { // Connection isConnected, isConnecting, connectionError, // State voiceState, isListening, isProcessing, isSpeaking, // Transcripts transcript, partialTranscript, // Audio isPlaying, audioLevel, // Tools activeToolCalls, // Metrics metrics, // Actions connect, disconnect, toggleVoice, bargeIn, } = useThinkerTalkerVoiceMode({ conversationId: "conv-123", voiceId: "TxGEqnHWrfWFTfGW9XjX", onTranscriptComplete: (text) => addMessage("user", text), onResponseComplete: (text) => addMessage("assistant", text), }); ``` ### Options ```typescript interface UseThinkerTalkerVoiceModeOptions { conversationId?: string; voiceId?: string; language?: string; bargeInEnabled?: boolean; autoConnect?: boolean; onTranscriptComplete?: (text: string) => void; onResponseDelta?: (delta: string) => void; onResponseComplete?: (text: string) => void; onToolCall?: (tool: TTToolCall) => void; onError?: (error: Error) => void; } ``` ### Return Values | Field | Type | Description | | ------------------- | --------------------- | ---------------------- | | `isConnected` | `boolean` | WebSocket connected | | `isConnecting` | `boolean` | Connection in progress | | `connectionError` | `Error \| null` | Connection error | | `voiceState` | `PipelineState` | Current state | | `isListening` | `boolean` | STT active | | `isProcessing` | `boolean` | LLM thinking | | `isSpeaking` | `boolean` | TTS playing | | `transcript` | `string` | Final transcript | | `partialTranscript` | `string` | Partial transcript | | `isPlaying` | `boolean` | Audio playing | | `audioLevel` | `number` | Mic level (0-1) | | `activeToolCalls` | `TTToolCall[]` | Current tools | | `metrics` | `TTVoiceMetrics` | Performance data | | `connect` | `() => Promise` | Start voice | | `disconnect` | `() => void` | End voice | | `toggleVoice` | `() => void` | Toggle on/off | | `bargeIn` | `() => void` | Interrupt | ## useVoicePreferencesSync Syncs voice settings with backend. ### Import ```typescript import { useVoicePreferencesSync } from "../hooks/useVoicePreferencesSync"; ``` ### Usage ```typescript const { preferences, isLoading, error, updatePreferences, resetToDefaults } = useVoicePreferencesSync(); // Update voice await updatePreferences({ voice_id: "21m00Tcm4TlvDq8ikWAM", // Rachel stability: 0.7, similarity_boost: 0.8, }); ``` ### Return Values | Field | Type | Description | | ------------------- | -------------------- | ---------------- | | `preferences` | `VoicePreferences` | Current settings | | `isLoading` | `boolean` | Loading state | | `error` | `Error \| null` | Last error | | `updatePreferences` | `(prefs) => Promise` | Save settings | | `resetToDefaults` | `() => Promise` | Reset all | ## Complete Example ```tsx import React, { useCallback } from "react"; import { useThinkerTalkerVoiceMode } from "../hooks/useThinkerTalkerVoiceMode"; import { useVoicePreferencesSync } from "../hooks/useVoicePreferencesSync"; function VoicePanel({ conversationId }: { conversationId: string }) { const { preferences } = useVoicePreferencesSync(); const { isConnected, isConnecting, voiceState, transcript, partialTranscript, activeToolCalls, metrics, connect, disconnect, bargeIn, } = useThinkerTalkerVoiceMode({ conversationId, voiceId: preferences.voice_id, onTranscriptComplete: useCallback((text) => { console.log("User said:", text); }, []), onResponseComplete: useCallback((text) => { console.log("AI said:", text); }, []), onToolCall: useCallback((tool) => { console.log("Tool called:", tool.name); }, []), }); return (
{/* Connection status */}
{isConnecting ? "Connecting..." : isConnected ? `Status: ${voiceState}` : "Disconnected"}
{/* Transcript display */}
{transcript || partialTranscript || "Listening..."}
{/* Tool calls */} {activeToolCalls.map((tool) => (
{tool.name}: {tool.status}
))} {/* Metrics */}
Latency: {metrics.totalLatencyMs}ms
{/* Controls */} {voiceState === "speaking" && }
); } ``` ## Error Handling ### Microphone Permission ```typescript // The hook detects permission errors if (status === "mic_permission_denied") { return (

Microphone access is required for voice mode.

); } ``` ### Connection Errors ```typescript const { error, status, reconnectAttempts } = useThinkerTalkerSession({ onError: (err) => { if (isMicPermissionError(err)) { showPermissionDialog(); } else { showErrorToast(err.message); } }, }); if (status === "reconnecting") { return
Reconnecting... (attempt {reconnectAttempts}/5)
; } if (status === "failed") { return
Connection failed. Please refresh.
; } ``` ## Performance Tips ### 1. Memoize Callbacks ```typescript const onTranscript = useCallback((t: TTTranscript) => { // Handle transcript }, []); const onAudioChunk = useCallback( (audio: string) => { playAudioChunk(base64ToArrayBuffer(audio)); }, [playAudioChunk], ); ``` ### 2. Avoid Re-renders ```typescript // Use refs for frequently updating values const metricsRef = useRef(metrics); useEffect(() => { metricsRef.current = metrics; }, [metrics]); ``` ### 3. Batch State Updates ```typescript // In the hook implementation const handleMessage = useCallback((msg) => { // React 18 automatically batches these setTranscript(msg.text); setPipelineState(msg.state); setMetrics((prev) => ({ ...prev, ...msg.metrics })); }, []); ``` ## Related Documentation - [Thinker-Talker Pipeline Overview](../THINKER_TALKER_PIPELINE.md) - [Voice Pipeline WebSocket API](../api-reference/voice-pipeline-ws.md) - [Voice Mode Settings Guide](../VOICE_MODE_SETTINGS_GUIDE.md) 6:["slug","frontend/thinker-talker-hooks","c"] 0:["X7oMT3VrOffzp0qvbeOas",[[["",{"children":["docs",{"children":[["slug","frontend/thinker-talker-hooks","c"],{"children":["__PAGE__?{\"slug\":[\"frontend\",\"thinker-talker-hooks\"]}",{}]}]}]},"$undefined","$undefined",true],["",{"children":["docs",{"children":[["slug","frontend/thinker-talker-hooks","c"],{"children":["__PAGE__",{},[["$L1",["$","div",null,{"children":[["$","div",null,{"className":"mb-6 flex items-center justify-between gap-4","children":[["$","div",null,{"children":[["$","p",null,{"className":"text-sm text-gray-500 dark:text-gray-400","children":"Docs / Raw"}],["$","h1",null,{"className":"text-3xl font-bold text-gray-900 dark:text-white","children":"Thinker-Talker Frontend Hooks"}],["$","p",null,{"className":"text-sm text-gray-600 dark:text-gray-400","children":["Sourced from"," ",["$","code",null,{"className":"font-mono text-xs","children":["docs/","frontend/thinker-talker-hooks.md"]}]]}]]}],["$","a",null,{"href":"https://github.com/mohammednazmy/VoiceAssist/edit/main/docs/frontend/thinker-talker-hooks.md","target":"_blank","rel":"noreferrer","className":"inline-flex items-center gap-2 rounded-md border border-gray-200 dark:border-gray-700 px-3 py-1.5 text-sm text-gray-700 dark:text-gray-200 hover:border-primary-500 dark:hover:border-primary-400 hover:text-primary-700 dark:hover:text-primary-300","children":"Edit on GitHub"}]]}],["$","div",null,{"className":"rounded-lg border border-gray-200 dark:border-gray-800 bg-white dark:bg-gray-900 p-6","children":["$","$L2",null,{"content":"$3"}]}],["$","div",null,{"className":"mt-6 flex flex-wrap gap-2 text-sm","children":[["$","$L4",null,{"href":"/reference/all-docs","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"← All documentation"}],["$","$L4",null,{"href":"/","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"Home"}]]}]]}],null],null],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children","$6","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/_next/static/css/7f586cdbbaa33ff7.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","className":"h-full","children":["$","body",null,{"className":"__className_f367f3 h-full bg-white dark:bg-gray-900","children":[["$","a",null,{"href":"#main-content","className":"skip-to-content","children":"Skip to main content"}],["$","$L8",null,{"children":[["$","$L9",null,{}],["$","$La",null,{}],["$","main",null,{"id":"main-content","className":"lg:pl-64","role":"main","aria-label":"Documentation content","children":["$","$Lb",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]]}]]}]}]],null],null],["$Lc",null]]]] c:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"Thinker-Talker Frontend Hooks | Docs | VoiceAssist Docs"}],["$","meta","3",{"name":"description","content":"React hooks for WebSocket connections, audio capture, and playback in the Thinker-Talker voice mode."}],["$","meta","4",{"name":"keywords","content":"VoiceAssist,documentation,medical AI,voice assistant,healthcare,HIPAA,API"}],["$","meta","5",{"name":"robots","content":"index, follow"}],["$","meta","6",{"name":"googlebot","content":"index, follow"}],["$","link","7",{"rel":"canonical","href":"https://assistdocs.asimo.io"}],["$","meta","8",{"property":"og:title","content":"VoiceAssist Documentation"}],["$","meta","9",{"property":"og:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","10",{"property":"og:url","content":"https://assistdocs.asimo.io"}],["$","meta","11",{"property":"og:site_name","content":"VoiceAssist Docs"}],["$","meta","12",{"property":"og:type","content":"website"}],["$","meta","13",{"name":"twitter:card","content":"summary"}],["$","meta","14",{"name":"twitter:title","content":"VoiceAssist Documentation"}],["$","meta","15",{"name":"twitter:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","16",{"name":"next-size-adjust"}]] 1:null