Thinker-Talker Voice Pipeline

Status: Production Ready Last Updated: 2025-12-01 Phase: Voice Pipeline Migration (Complete)

Overview

The Thinker-Talker (T/T) pipeline is VoiceAssist's voice processing architecture that replaces the OpenAI Realtime API with a local orchestration approach. It provides unified conversation context, full tool/RAG support, and custom TTS with ElevenLabs for superior voice quality.

┌─────────────────────────────────────────────────────────────────────────────┐
│                        Thinker-Talker Pipeline                               │
│                                                                              │
│   ┌──────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────┐     │
│   │  Audio   │───>│ Deepgram STT │───>│ GPT-4o       │───>│ElevenLabs│     │
│   │  Input   │    │ (Streaming)  │    │ Thinker      │    │   TTS    │     │
│   └──────────┘    └──────────────┘    └──────────────┘    └──────────┘     │
│        │                │                    │                  │           │
│        │           Transcripts          Tool Calls         Audio Out        │
│        │                │                    │                  │           │
│        v                v                    v                  v           │
│   ┌─────────────────────────────────────────────────────────────────┐      │
│   │                    WebSocket Handler                             │      │
│   │              (Bidirectional Client Communication)                │      │
│   └─────────────────────────────────────────────────────────────────┘      │
└─────────────────────────────────────────────────────────────────────────────┘

Benefits Over OpenAI Realtime API

Architecture Components

1. Voice Pipeline Service

Location: services/api-gateway/app/services/voice_pipeline_service.py

Orchestrates the complete STT → Thinker → Talker flow:

class VoicePipelineService:
    """
    Orchestrates the complete voice pipeline:
    1. Receive audio from client
    2. Stream to Deepgram STT
    3. Send transcripts to Thinker (LLM)
    4. Stream response tokens to Talker (TTS)
    5. Send audio chunks back to client
    """

Configuration:

@dataclass
class PipelineConfig:
    # STT Settings
    stt_language: str = "en"
    stt_sample_rate: int = 16000
    stt_endpointing_ms: int = 800    # Wait for natural pauses
    stt_utterance_end_ms: int = 1500  # Finalize after 1.5s silence

    # TTS Settings - defaults from voice_constants.py (single source of truth)
    # See docs/voice/voice-configuration.md for details
    voice_id: str = DEFAULT_VOICE_ID  # Brian (from voice_constants.py)
    tts_model: str = DEFAULT_TTS_MODEL  # eleven_flash_v2_5

    # Barge-in
    barge_in_enabled: bool = True

2. Thinker Service

Location: services/api-gateway/app/services/thinker_service.py

The reasoning engine that processes transcribed speech:

class ThinkerService:
    """
    Unified reasoning service for the Thinker/Talker pipeline.

    Handles:
    - Conversation context management (persisted across turns)
    - Streaming LLM responses with token callbacks
    - Tool calling with result injection
    - Cancellation support
    """

Key Features:

• ConversationContext: Maintains history (max 20 messages) with smart trimming
• Tool Registry: Supports calendar, search, medical calculators, KB search
• Streaming: Token-by-token callbacks for low-latency TTS
• State Machine: IDLE → PROCESSING → TOOL_CALLING → GENERATING → COMPLETE

3. Talker Service

Location: services/api-gateway/app/services/talker_service.py

Text-to-Speech synthesis with streaming audio:

class TalkerService:
    """
    Unified TTS service for the Thinker/Talker pipeline.

    Handles:
    - Streaming LLM tokens through sentence chunker
    - Audio queue management for gapless playback
    - Cancellation (barge-in support)
    """

Voice Configuration:

@dataclass
class VoiceConfig:
    provider: TTSProvider = TTSProvider.ELEVENLABS
    voice_id: str = "TxGEqnHWrfWFTfGW9XjX"  # Josh
    model_id: str = "eleven_turbo_v2_5"
    stability: float = 0.78       # Voice consistency
    similarity_boost: float = 0.85  # Voice clarity
    style: float = 0.08           # Natural, less dramatic
    output_format: str = "pcm_24000"  # Low-latency streaming

4. Sentence Chunker

Location: services/api-gateway/app/services/sentence_chunker.py

Optimizes LLM output for TTS with low latency:

class SentenceChunker:
    """
    Low-latency phrase chunker for TTS processing.

    Strategy:
    - Primary: Split on sentence boundaries (. ! ?)
    - Secondary: Split on clause boundaries (, ; :) after min chars
    - Emergency: Force split at max chars

    Config (optimized for speed):
    - min_chunk_chars: 40   (avoid tiny fragments)
    - optimal_chunk_chars: 120  (natural phrases)
    - max_chunk_chars: 200  (force split)
    """

5. WebSocket Handler

Location: services/api-gateway/app/services/thinker_talker_websocket_handler.py

Manages bidirectional client communication:

class ThinkerTalkerWebSocketHandler:
    """
    WebSocket handler for Thinker/Talker voice pipeline.

    Protocol Messages (Client → Server):
    - audio.input: Base64 PCM16 audio
    - audio.input.complete: Signal end of speech
    - barge_in: Interrupt AI response
    - voice.mode: Activate/deactivate voice mode

    Protocol Messages (Server → Client):
    - transcript.delta/complete: STT results
    - response.delta/complete: LLM response
    - audio.output: TTS audio chunk
    - tool.call/result: Tool execution
    - voice.state: Pipeline state update
    """

Data Flow

Complete Request/Response Cycle

1. User speaks into microphone
   │
   ▼
2. Frontend captures PCM16 audio (16kHz)
   │
   ▼
3. Audio streamed via WebSocket (audio.input messages)
   │
   ▼
4. Deepgram STT processes audio stream
   │
   ├──> transcript.delta (partial text)
   │
   └──> transcript.complete (final text)
        │
        ▼
5. ThinkerService receives transcript
   │
   ├──> Adds to ConversationContext
   │
   ├──> Calls GPT-4o with tools
   │
   ├──> If tool call needed:
   │    │
   │    ├──> tool.call sent to client
   │    │
   │    ├──> Tool executed
   │    │
   │    └──> tool.result sent to client
   │
   └──> response.delta (streaming tokens)
        │
        ▼
6. TalkerService receives tokens
   │
   ├──> SentenceChunker buffers tokens
   │
   ├──> Complete sentences → ElevenLabs TTS
   │
   └──> audio.output (streaming PCM)
        │
        ▼
7. Frontend plays audio via Web Audio API

Barge-in Flow

1. AI is speaking (audio.output streaming)
   │
2. User starts speaking
   │
   ▼
3. Frontend sends barge_in message
   │
   ▼
4. Backend:
   ├──> Cancels TalkerSession
   ├──> Clears audio queue
   └──> Resets pipeline to LISTENING
   │
   ▼
5. New user speech processed normally

State Machine

                    ┌─────────────────┐
                    │      IDLE       │
                    │  (waiting for   │
                    │   user input)   │
                    └────────┬────────┘
                             │
                    audio.input received
                             │
                             ▼
                    ┌─────────────────┐
                    │   LISTENING     │
                    │  (STT active,   │
                    │  collecting)    │
                    └────────┬────────┘
                             │
                    transcript.complete
                             │
                             ▼
                    ┌─────────────────┐
                    │  PROCESSING     │◄─────────┐
                    │  (LLM thinking) │          │
                    └────────┬────────┘          │
                             │                   │
              ┌──────────────┼──────────────┐    │
              │              │              │    │
         tool_call     no tools      error  │    │
              │              │              │    │
              ▼              ▼              │    │
    ┌─────────────────┐ ┌──────────┐       │    │
    │  TOOL_CALLING   │ │GENERATING│       │    │
    │  (executing     │ │(streaming│       │    │
    │   tool)         │ │ response)│       │    │
    └────────┬────────┘ └────┬─────┘       │    │
             │               │             │    │
        tool_result     response.complete  │    │
             │               │             │    │
             └───────┬───────┘             │    │
                     │                     │    │
                     ▼                     │    │
            ┌─────────────────┐            │    │
            │    SPEAKING     │            │    │
            │  (TTS playing)  │────────────┘    │
            └────────┬────────┘  (more to say)  │
                     │                          │
           audio complete or barge_in           │
                     │                          │
                     ▼                          │
            ┌─────────────────┐                 │
            │   CANCELLED     │─────────────────┘
            │  (interrupted)  │   (restart listening)
            └─────────────────┘

WebSocket Protocol

Client → Server Messages

Server → Client Messages

Frontend Integration

useThinkerTalkerSession Hook

Location: apps/web-app/src/hooks/useThinkerTalkerSession.ts

const {
  status, // 'disconnected' | 'connecting' | 'ready' | 'error'
  pipelineState, // 'idle' | 'listening' | 'processing' | 'speaking'
  transcript, // Final user transcript
  metrics, // Latency and usage metrics
  connect, // Start session
  disconnect, // End session
  sendAudio, // Send audio chunk
  bargeIn, // Interrupt AI
} = useThinkerTalkerSession({
  conversation_id: "...",
  voiceSettings: {
    voice_id: "TxGEqnHWrfWFTfGW9XjX",
    language: "en",
    barge_in_enabled: true,
  },
  onTranscript: (t) => console.log("Transcript:", t),
  onAudioChunk: (audio) => playAudio(audio),
  onToolCall: (tool) => console.log("Tool:", tool),
});

useTTAudioPlayback Hook

Location: apps/web-app/src/hooks/useTTAudioPlayback.ts

Handles streaming audio playback with barge-in support:

const {
  isPlaying,
  queuedChunks,
  playAudioChunk, // Add chunk to queue
  stopPlayback, // Cancel playback (barge-in)
  clearQueue, // Clear pending audio
} = useTTAudioPlayback({
  sampleRate: 24000,
  onPlaybackEnd: () => console.log("Playback complete"),
});

Configuration Reference

Backend Environment Variables

# LLM Settings
MODEL_SELECTION_DEFAULT=gpt-4o
OPENAI_API_KEY=sk-...
OPENAI_TIMEOUT_SEC=30

# TTS Settings
ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_ID=TxGEqnHWrfWFTfGW9XjX
ELEVENLABS_MODEL_ID=eleven_turbo_v2_5

# STT Settings
DEEPGRAM_API_KEY=...

Voice Configuration Options

Available ElevenLabs Voices

Metrics & Observability

TTVoiceMetrics

interface TTVoiceMetrics {
  connectionTimeMs: number; // Connect to ready
  sttLatencyMs: number; // Speech end to transcript
  llmFirstTokenMs: number; // Transcript to first token
  ttsFirstAudioMs: number; // First token to first audio
  totalLatencyMs: number; // Speech end to first audio
  userUtteranceCount: number;
  aiResponseCount: number;
  toolCallCount: number;
  bargeInCount: number;
}

Latency Targets

Troubleshooting

Common Issues

1. No audio output

• Check ElevenLabs API key is valid
• Verify voice_id exists in available voices
• Check browser audio permissions

2. High latency

• Check network connection
• Verify STT endpoint is responsive
• Consider reducing chunk sizes

3. Barge-in not working

• Ensure barge_in_enabled: true in config
• Check WebSocket connection is stable
• Verify frontend is sending barge_in message

4. Tool calls failing

• Check user authentication (user_id required)
• Verify tool is registered in ToolRegistry
• Check tool-specific API keys (calendar, etc.)

Debug Logging

Enable verbose logging:

# Backend
import logging
logging.getLogger("app.services.thinker_service").setLevel(logging.DEBUG)
logging.getLogger("app.services.talker_service").setLevel(logging.DEBUG)

// Frontend
import { voiceLog } from "../lib/logger";
voiceLog.setLevel("debug");

Related Documentation

• Thinker Service API
• Talker Service API
• Voice Pipeline WebSocket Protocol
• Frontend Voice Hooks
• Voice Mode Settings Guide

Changelog

2025-12-01 - Initial Release

• Complete Thinker-Talker pipeline implementation
• Deepgram STT integration with streaming
• ElevenLabs TTS with sentence chunking
• Full tool calling support
• Barge-in capability
• Unified conversation context with chat mode

Voice Mode Pipeline

Status: Production-ready Last Updated: 2025-12-03

This document describes the unified Voice Mode pipeline architecture, data flow, metrics, and testing strategy. It serves as the canonical reference for developers working on real-time voice features.

Voice Pipeline Modes

VoiceAssist supports two voice pipeline modes:

Thinker-Talker Pipeline (Primary)

The Thinker-Talker pipeline is the recommended approach, providing:

• Unified conversation context between voice and chat modes
• Full tool/RAG support in voice interactions
• Custom TTS via ElevenLabs with premium voices
• Lower cost per interaction

Documentation: THINKER_TALKER_PIPELINE.md

[Audio] → [Deepgram STT] → [GPT-4o Thinker] → [ElevenLabs TTS] → [Audio Out]
              │                    │                    │
         Transcripts          Tool Calls           Audio Chunks
              │                    │                    │
              └───────── WebSocket Handler ──────────────┘

OpenAI Realtime API (Legacy)

The original implementation using OpenAI's Realtime API directly. Still supported for backward compatibility.

Implementation Status

Thinker-Talker Components

OpenAI Realtime Components (Legacy)

Full status: See Implementation Status for all components.

Overview

Voice Mode enables real-time voice conversations with the AI assistant using OpenAI's Realtime API. The pipeline handles:

• Ephemeral session authentication (no raw API keys in browser)
• WebSocket-based bidirectional voice streaming
• Voice activity detection (VAD) with user-configurable sensitivity
• User settings propagation (voice, language, VAD threshold)
• Chat timeline integration (voice messages appear in chat)
• Connection state management with automatic reconnection
• Barge-in support (interrupt AI while speaking)
• Audio playback management (prevent overlapping responses)
• Metrics tracking for observability

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                              FRONTEND                                        │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────────┐     ┌─────────────────────┐     ┌───────────────┐  │
│  │  VoiceModePanel     │────▶│useRealtimeVoice     │────▶│ voiceSettings │  │
│  │  (UI Component)     │     │Session (Hook)       │     │ Store         │  │
│  │  - Start/Stop       │     │- connect()          │     │ - voice       │  │
│  │  - Status display   │     │- disconnect()       │     │ - language    │  │
│  │  - Metrics logging  │     │- sendMessage()      │     │ - vadSens     │  │
│  └─────────┬───────────┘     └──────────┬──────────┘     └───────────────┘  │
│            │                            │                                    │
│            │                            │ onUserMessage()/onAssistantMessage()
│            │                            ▼                                    │
│  ┌─────────▼───────────┐     ┌─────────────────────┐                        │
│  │  MessageInput       │     │  ChatPage           │                        │
│  │  - Voice toggle     │────▶│  - useChatSession   │                        │
│  │  - Panel container  │     │  - addMessage()     │                        │
│  └─────────────────────┘     └─────────────────────┘                        │
│                                                                              │
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       │
                                       │ POST /api/voice/realtime-session
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              BACKEND                                         │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────────┐     ┌─────────────────────┐                        │
│  │  voice.py           │────▶│  realtime_voice_    │                        │
│  │  (FastAPI Router)   │     │  service.py         │                        │
│  │  - /realtime-session│     │  - generate_session │                        │
│  │  - Timing logs      │     │  - ephemeral token  │                        │
│  └─────────────────────┘     └──────────┬──────────┘                        │
│                                         │                                    │
│                                         │ POST /v1/realtime/sessions         │
│                                         ▼                                    │
│                              ┌─────────────────────┐                        │
│                              │  OpenAI API         │                        │
│                              │  - Ephemeral token  │                        │
│                              │  - Voice config     │                        │
│                              └─────────────────────┘                        │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                       │
                                       │ WebSocket wss://api.openai.com/v1/realtime
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          OPENAI REALTIME API                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│  - Server-side VAD (voice activity detection)                                │
│  - Bidirectional audio streaming (PCM16)                                     │
│  - Real-time transcription (Whisper)                                         │
│  - GPT-4o responses with audio synthesis                                     │
└─────────────────────────────────────────────────────────────────────────────┘

Backend: /api/voice/realtime-session

Location: services/api-gateway/app/api/voice.py

Request

interface RealtimeSessionRequest {
  conversation_id?: string; // Optional conversation context
  voice?: string; // "alloy" | "echo" | "fable" | "onyx" | "nova" | "shimmer"
  language?: string; // "en" | "es" | "fr" | "de" | "it" | "pt"
  vad_sensitivity?: number; // 0-100 (maps to threshold: 0→0.9, 100→0.1)
}

Response

interface RealtimeSessionResponse {
  url: string; // WebSocket URL: "wss://api.openai.com/v1/realtime"
  model: string; // "gpt-4o-realtime-preview"
  session_id: string; // Unique session identifier
  expires_at: number; // Unix timestamp (epoch seconds)
  conversation_id: string | null;
  auth: {
    type: "ephemeral_token";
    token: string; // Ephemeral token (ek_...), NOT raw API key
    expires_at: number; // Token expiry (5 minutes)
  };
  voice_config: {
    voice: string; // Selected voice
    modalities: ["text", "audio"];
    input_audio_format: "pcm16";
    output_audio_format: "pcm16";
    input_audio_transcription: { model: "whisper-1" };
    turn_detection: {
      type: "server_vad";
      threshold: number; // 0.1 (sensitive) to 0.9 (insensitive)
      prefix_padding_ms: number;
      silence_duration_ms: number;
    };
  };
}

VAD Sensitivity Mapping

The frontend uses a 0-100 scale for user-friendly VAD sensitivity:

Formula: threshold = 0.9 - (vad_sensitivity / 100 * 0.8)

Observability

Backend logs timing and context for each session request:

# Request logging
logger.info(
    f"Creating Realtime session for user {current_user.id}",
    extra={
        "user_id": current_user.id,
        "conversation_id": request.conversation_id,
        "voice": request.voice,
        "language": request.language,
        "vad_sensitivity": request.vad_sensitivity,
    },
)

# Success logging with duration
duration_ms = int((time.monotonic() - start_time) * 1000)
logger.info(
    f"Realtime session created for user {current_user.id}",
    extra={
        "user_id": current_user.id,
        "session_id": config["session_id"],
        "voice": config.get("voice_config", {}).get("voice"),
        "duration_ms": duration_ms,
    },
)

Frontend Hook: useRealtimeVoiceSession

Location: apps/web-app/src/hooks/useRealtimeVoiceSession.ts

Usage

const {
  status, // 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed' | 'expired' | 'error'
  transcript, // Current transcript text
  isSpeaking, // Is the AI currently speaking?
  isConnected, // Derived: status === 'connected'
  isConnecting, // Derived: status === 'connecting' || 'reconnecting'
  canSend, // Can send messages?
  error, // Error message if any
  metrics, // VoiceMetrics object
  connect, // () => Promise<void> - start session
  disconnect, // () => void - end session
  sendMessage, // (text: string) => void - send text message
} = useRealtimeVoiceSession({
  conversationId,
  voice, // From voiceSettingsStore
  language, // From voiceSettingsStore
  vadSensitivity, // From voiceSettingsStore (0-100)
  onConnected, // Callback when connected
  onDisconnected, // Callback when disconnected
  onError, // Callback on error
  onUserMessage, // Callback with user transcript
  onAssistantMessage, // Callback with AI response
  onMetricsUpdate, // Callback when metrics change
});

Connection States

disconnected ──▶ connecting ──▶ connected
                      │              │
                      ▼              ▼
                   failed ◀──── reconnecting
                      │              │
                      ▼              ▼
                  expired ◀────── error

WebSocket Connection

The hook connects using three protocols for authentication:

const ws = new WebSocket(url, ["realtime", "openai-beta.realtime-v1", `openai-insecure-api-key.${ephemeralToken}`]);

Voice Settings Store

Location: apps/web-app/src/stores/voiceSettingsStore.ts

Schema

interface VoiceSettings {
  voice: "alloy" | "echo" | "fable" | "onyx" | "nova" | "shimmer";
  language: "en" | "es" | "fr" | "de" | "it" | "pt";
  vadSensitivity: number; // 0-100
  autoStartOnOpen: boolean; // Auto-start voice when panel opens
  showStatusHints: boolean; // Show helper text in UI
}

Persistence

Settings are persisted to localStorage under key voiceassist-voice-settings using Zustand's persist middleware.

Defaults

Chat Integration

Location: apps/web-app/src/pages/ChatPage.tsx

Message Flow

1. User speaks → VoiceModePanel receives final transcript
2. VoiceModePanel calls onUserMessage(transcript)
3. ChatPage receives callback, calls useChatSession.addMessage()
4. Message added to timeline with metadata: { source: "voice" }

// ChatPage.tsx
const handleVoiceUserMessage = (content: string) => {
  addMessage({
    role: "user",
    content,
    metadata: { source: "voice" },
  });
};

const handleVoiceAssistantMessage = (content: string) => {
  addMessage({
    role: "assistant",
    content,
    metadata: { source: "voice" },
  });
};

Message Structure

interface VoiceMessage {
  id: string; // "voice-{timestamp}-{random}"
  role: "user" | "assistant";
  content: string;
  timestamp: number;
  metadata: {
    source: "voice"; // Distinguishes from text messages
  };
}

Barge-in & Audio Playback

Location: apps/web-app/src/components/voice/VoiceModePanel.tsx, apps/web-app/src/hooks/useRealtimeVoiceSession.ts

Barge-in Flow

When the user starts speaking while the AI is responding, the system immediately:

1. Detects speech start via OpenAI's input_audio_buffer.speech_started event
2. Cancels active response by sending response.cancel to OpenAI
3. Stops audio playback via onSpeechStarted callback
4. Clears pending responses to prevent stale audio from playing

User speaks → speech_started event → response.cancel → stopCurrentAudio()
                                                            ↓
                                                    Audio stops
                                                    Queue cleared
                                                    Response ID incremented

Response Cancellation

Location: useRealtimeVoiceSession.ts - handleRealtimeMessage

case "input_audio_buffer.speech_started":
  setIsSpeaking(true);
  setPartialTranscript("");

  // Barge-in: Cancel any active response when user starts speaking
  if (activeResponseIdRef.current && wsRef.current?.readyState === WebSocket.OPEN) {
    wsRef.current.send(JSON.stringify({ type: "response.cancel" }));
    activeResponseIdRef.current = null;
  }

  // Notify parent to stop audio playback
  options.onSpeechStarted?.();
  break;

Audio Playback Management

Location: VoiceModePanel.tsx

The panel tracks audio playback state to prevent overlapping responses:

// Track currently playing Audio element
const currentAudioRef = useRef<HTMLAudioElement | null>(null);

// Prevent overlapping response processing
const isProcessingResponseRef = useRef(false);

// Response ID to invalidate stale responses after barge-in
const currentResponseIdRef = useRef<number>(0);

Stop current audio function:

const stopCurrentAudio = useCallback(() => {
  if (currentAudioRef.current) {
    currentAudioRef.current.pause();
    currentAudioRef.current.currentTime = 0;
    if (currentAudioRef.current.src.startsWith("blob:")) {
      URL.revokeObjectURL(currentAudioRef.current.src);
    }
    currentAudioRef.current = null;
  }
  audioQueueRef.current = [];
  isPlayingRef.current = false;
  currentResponseIdRef.current++; // Invalidate pending responses
  isProcessingResponseRef.current = false;
}, []);

Overlap Prevention

When a relay result arrives, the handler checks:

1. Already processing? Skip if isProcessingResponseRef.current === true
2. Response ID valid? Skip playback if ID changed (barge-in occurred)

onRelayResult: async ({ answer }) => {
  if (answer) {
    // Prevent overlapping responses
    if (isProcessingResponseRef.current) {
      console.log("[VoiceModePanel] Skipping response - already processing another");
      return;
    }

    const responseId = ++currentResponseIdRef.current;
    isProcessingResponseRef.current = true;

    // ... synthesis and playback ...

    // Check if response is still valid before playback
    if (responseId !== currentResponseIdRef.current) {
      console.log("[VoiceModePanel] Response cancelled - skipping playback");
      return;
    }
  }
};

Error Handling

Benign cancellation errors (e.g., "Cancellation failed: no active response found") are handled gracefully:

case "error": {
  const errorMessage = message.error?.message || "Realtime API error";

  // Ignore benign cancellation errors
  if (
    errorMessage.includes("Cancellation failed") ||
    errorMessage.includes("no active response")
  ) {
    voiceLog.debug(`Ignoring benign error: ${errorMessage}`);
    break;
  }

  handleError(new Error(errorMessage));
  break;
}

Metrics

Location: apps/web-app/src/hooks/useRealtimeVoiceSession.ts

VoiceMetrics Interface

interface VoiceMetrics {
  connectionTimeMs: number | null; // Time to establish connection
  timeToFirstTranscriptMs: number | null; // Time to first user transcript
  lastSttLatencyMs: number | null; // Speech-to-text latency
  lastResponseLatencyMs: number | null; // AI response latency
  sessionDurationMs: number | null; // Total session duration
  userTranscriptCount: number; // Number of user turns
  aiResponseCount: number; // Number of AI turns
  reconnectCount: number; // Number of reconnections
  sessionStartedAt: number | null; // Session start timestamp
}

Frontend Logging

VoiceModePanel logs key metrics to console:

// Connection time
console.log(`[VoiceModePanel] voice_session_connect_ms=${metrics.connectionTimeMs}`);

// STT latency
console.log(`[VoiceModePanel] voice_stt_latency_ms=${metrics.lastSttLatencyMs}`);

// Response latency
console.log(`[VoiceModePanel] voice_first_reply_ms=${metrics.lastResponseLatencyMs}`);

// Session duration
console.log(`[VoiceModePanel] voice_session_duration_ms=${metrics.sessionDurationMs}`);

Consuming Metrics

Developers can plug into metrics via the onMetricsUpdate callback:

useRealtimeVoiceSession({
  onMetricsUpdate: (metrics) => {
    // Send to telemetry service
    analytics.track("voice_session_metrics", {
      connection_ms: metrics.connectionTimeMs,
      stt_latency_ms: metrics.lastSttLatencyMs,
      response_latency_ms: metrics.lastResponseLatencyMs,
      duration_ms: metrics.sessionDurationMs,
    });
  },
});

Metrics Export to Backend

Metrics can be automatically exported to the backend for aggregation and alerting.

Backend Endpoint: POST /api/voice/metrics

Location: services/api-gateway/app/api/voice.py

Request Schema

interface VoiceMetricsPayload {
  conversation_id?: string;
  connection_time_ms?: number;
  time_to_first_transcript_ms?: number;
  last_stt_latency_ms?: number;
  last_response_latency_ms?: number;
  session_duration_ms?: number;
  user_transcript_count: number;
  ai_response_count: number;
  reconnect_count: number;
  session_started_at?: number;
}

Response

interface VoiceMetricsResponse {
  status: "ok";
}

Privacy

No PHI or transcript content is sent. Only timing metrics and counts.

Frontend Configuration

Metrics export is controlled by environment variables:

• Production (import.meta.env.PROD): Metrics sent automatically
• Development: Set VITE_ENABLE_VOICE_METRICS=true to enable

The export uses navigator.sendBeacon() for reliability (survives page navigation).

Backend Logging

Metrics are logged with user context:

logger.info(
    "VoiceMetrics received",
    extra={
        "user_id": current_user.id,
        "conversation_id": payload.conversation_id,
        "connection_time_ms": payload.connection_time_ms,
        "session_duration_ms": payload.session_duration_ms,
        ...
    },
)

Testing

# Backend
cd /home/asimo/VoiceAssist/services/api-gateway
source venv/bin/activate && export PYTHONPATH=.
python -m pytest tests/integration/test_voice_metrics.py -v

Security

Ephemeral Token Architecture

CRITICAL: The browser NEVER receives the raw OpenAI API key.

1. Backend holds OPENAI_API_KEY securely
2. Frontend requests session via /api/voice/realtime-session
3. Backend creates ephemeral token via OpenAI /v1/realtime/sessions
4. Ephemeral token returned to frontend (valid ~5 minutes)
5. Frontend connects WebSocket using ephemeral token

Token Refresh

The hook monitors session.expires_at and can trigger refresh before expiry. If the token expires mid-session, status transitions to expired.

Testing

Voice Pipeline Smoke Suite

Run these commands to validate the voice pipeline:

# 1. Backend tests (CI-safe, mocked)
cd /home/asimo/VoiceAssist/services/api-gateway
source venv/bin/activate
export PYTHONPATH=.
python -m pytest tests/integration/test_openai_config.py -v

# 2. Frontend unit tests (run individually to avoid OOM)
cd /home/asimo/VoiceAssist/apps/web-app
export NODE_OPTIONS="--max-old-space-size=768"

npx vitest run src/hooks/__tests__/useRealtimeVoiceSession.test.ts --reporter=dot
npx vitest run src/hooks/__tests__/useChatSession-voice-integration.test.ts --reporter=dot
npx vitest run src/stores/__tests__/voiceSettingsStore.test.ts --reporter=dot
npx vitest run src/components/voice/__tests__/VoiceModeSettings.test.tsx --reporter=dot
npx vitest run src/components/chat/__tests__/MessageInput-voice-settings.test.tsx --reporter=dot

# 3. E2E tests (Chromium, mocked backend)
cd /home/asimo/VoiceAssist
npx playwright test \
  e2e/voice-mode-navigation.spec.ts \
  e2e/voice-mode-session-smoke.spec.ts \
  e2e/voice-mode-voice-chat-integration.spec.ts \
  --project=chromium --reporter=list

Test Coverage Summary

Total: 95 tests

Live Testing

To test with real OpenAI backend:

# Backend (requires OPENAI_API_KEY in .env)
LIVE_REALTIME_TESTS=1 python -m pytest tests/integration/test_openai_config.py -v

# E2E (requires running backend + valid API key)
LIVE_REALTIME_E2E=1 npx playwright test e2e/voice-mode-session-smoke.spec.ts

File Reference

Backend

Frontend

Tests

Related Documentation

• VOICE_MODE_ENHANCEMENT_10_PHASE.md - 10-phase enhancement plan (emotion, dictation, analytics)
• VOICE_MODE_SETTINGS_GUIDE.md - User settings configuration
• TESTING_GUIDE.md - E2E testing strategy and validation checklist

Observability & Monitoring (Phase 3)

Implemented: 2025-12-02

The voice pipeline includes comprehensive observability features for production monitoring.

Error Taxonomy (voice_errors.py)

Location: services/api-gateway/app/core/voice_errors.py

Structured error classification with 8 categories and 40+ error codes:

Each error code includes:

• Recoverability flag (can auto-retry)
• Retry configuration (delay, max attempts)
• User-friendly description

Voice Metrics (metrics.py)

Location: services/api-gateway/app/core/metrics.py

Prometheus metrics for voice pipeline monitoring:

Per-Stage Latency Tracking (voice_timing.py)

Location: services/api-gateway/app/core/voice_timing.py

Pipeline stages tracked:

• audio_receive - Time to receive audio from client
• vad_process - Voice activity detection time
• stt_transcribe - Speech-to-text latency
• llm_process - LLM inference time
• tts_synthesize - Text-to-speech synthesis
• audio_send - Time to send audio to client
• ttfa - Time to first audio (end-to-end)

Usage:

from app.core.voice_timing import create_pipeline_timings, PipelineStage

timings = create_pipeline_timings(session_id="abc123")

with timings.time_stage(PipelineStage.STT_TRANSCRIBE):
    transcript = await stt_client.transcribe(audio)

timings.record_ttfa()  # When first audio byte ready
timings.finalize()     # When response complete

SLO Alerts (voice_slo_alerts.yml)

Location: infrastructure/observability/prometheus/rules/voice_slo_alerts.yml

SLO targets with Prometheus alerting rules:

Client Telemetry (voiceTelemetry.ts)

Location: apps/web-app/src/lib/voiceTelemetry.ts

Frontend telemetry with:

• Network quality assessment via Network Information API
• Browser performance metrics via Performance.memory API
• Jitter estimation for network quality
• Batched reporting (10s intervals)
• Beacon API for reliable delivery on page unload

import { getVoiceTelemetry } from "@/lib/voiceTelemetry";

const telemetry = getVoiceTelemetry();
telemetry.startSession(sessionId);
telemetry.recordLatency("stt", 150);
telemetry.recordLatency("ttfa", 180);
telemetry.endSession();

Voice Health Endpoint (/health/voice)

Location: services/api-gateway/app/api/health.py

Comprehensive voice subsystem health check:

curl https://assist.asimo.io/health/voice

Response:

{
  "status": "healthy",
  "providers": {
    "openai": { "status": "up", "latency_ms": 120.5 },
    "elevenlabs": { "status": "up", "latency_ms": 85.2 },
    "deepgram": { "status": "up", "latency_ms": 95.8 }
  },
  "session_store": { "status": "up", "active_sessions": 5 },
  "metrics": { "active_sessions": 5 },
  "slo": { "ttfa_target_ms": 200, "error_rate_target": 0.01 }
}

Debug Logging Configuration

Location: services/api-gateway/app/core/logging.py

Configurable voice log verbosity via VOICE_LOG_LEVEL environment variable:

Usage:

from app.core.logging import get_voice_logger

voice_log = get_voice_logger(__name__)
voice_log.session_start(session_id="abc123", provider="thinker_talker")
voice_log.latency("stt_transcribe", 150.5, session_id="abc123")
voice_log.error("voice_connection_failed", error_code="CONN_001")

Phase 9: Offline & Network Fallback

Implemented: 2025-12-03

The voice pipeline now includes comprehensive offline support and network-aware fallback mechanisms.

Network Monitoring (networkMonitor.ts)

Location: apps/web-app/src/lib/offline/networkMonitor.ts

Continuously monitors network health using multiple signals:

• Navigator.onLine: Basic online/offline detection
• Network Information API: Connection type, downlink speed, RTT
• Health Check Pinging: Periodic /api/health pings for latency measurement

import { getNetworkMonitor } from "@/lib/offline/networkMonitor";

const monitor = getNetworkMonitor();
monitor.subscribe((status) => {
  console.log(`Network quality: ${status.quality}`);
  console.log(`Health check latency: ${status.healthCheckLatencyMs}ms`);
});

Network Quality Levels

Configuration

const monitor = createNetworkMonitor({
  healthCheckUrl: "/api/health",
  healthCheckIntervalMs: 30000, // 30 seconds
  healthCheckTimeoutMs: 5000, // 5 seconds
  goodLatencyThresholdMs: 100,
  moderateLatencyThresholdMs: 200,
  poorLatencyThresholdMs: 500,
  failuresBeforeUnhealthy: 3,
});

useNetworkStatus Hook

Location: apps/web-app/src/hooks/useNetworkStatus.ts

React hook providing network status with computed properties:

const {
  isOnline,
  isHealthy,
  quality,
  healthCheckLatencyMs,
  effectiveType, // "4g", "3g", "2g", "slow-2g"
  downlink, // Mbps
  rtt, // Round-trip time ms
  isSuitableForVoice, // quality >= "good" && isHealthy
  shouldUseOffline, // !isOnline || !isHealthy || quality < "moderate"
  qualityScore, // 0-4 (offline=0, poor=1, moderate=2, good=3, excellent=4)
  checkNow, // Force immediate health check
} = useNetworkStatus();

Offline VAD with Network Fallback

Location: apps/web-app/src/hooks/useOfflineVAD.ts

The useOfflineVADWithFallback hook automatically switches between network and offline VAD:

const {
  isListening,
  isSpeaking,
  currentEnergy,
  isUsingOfflineVAD, // Currently using offline mode?
  networkAvailable,
  networkQuality,
  modeReason, // "network_vad" | "network_unavailable" | "poor_quality" | "forced_offline"
  forceOffline, // Manually switch to offline
  forceNetwork, // Manually switch to network (if available)
  startListening,
  stopListening,
} = useOfflineVADWithFallback({
  useNetworkMonitor: true,
  minNetworkQuality: "moderate",
  networkRecoveryDelayMs: 2000, // Prevent flapping
  onFallbackToOffline: () => console.log("Switched to offline VAD"),
  onReturnToNetwork: () => console.log("Returned to network VAD"),
});

Fallback Decision Flow

┌────────────────────┐
│  Network Monitor   │
│  Health Check      │
└─────────┬──────────┘
          │
          ▼
┌────────────────────┐     NO     ┌────────────────────┐
│  Is Online?        │──────────▶│  Use Offline VAD   │
└─────────┬──────────┘            └────────────────────┘
          │ YES
          ▼
┌────────────────────┐     NO     ┌────────────────────┐
│  Is Healthy?       │──────────▶│  Use Offline VAD   │
│  (3+ checks pass)  │            │  reason: unhealthy │
└─────────┬──────────┘            └────────────────────┘
          │ YES
          ▼
┌────────────────────┐     NO     ┌────────────────────┐
│  Quality ≥ Min?    │──────────▶│  Use Offline VAD   │
│  (e.g., moderate)  │            │  reason: poor_qual │
└─────────┬──────────┘            └────────────────────┘
          │ YES
          ▼
┌────────────────────┐
│  Use Network VAD   │
│  (cloud processing)│
└────────────────────┘

TTS Caching (useTTSCache)

Location: apps/web-app/src/hooks/useOfflineVAD.ts

Caches synthesized TTS audio for offline playback:

const {
  getTTS, // Get audio (from cache or fresh)
  preload, // Preload common phrases
  isCached, // Check if text is cached
  stats, // { entryCount, sizeMB, hitRate }
  clear, // Clear cache
} = useTTSCache({
  voice: "alloy",
  maxSizeMB: 50,
  ttsFunction: async (text) => synthesizeAudio(text),
});

// Preload common phrases on app start
await preload(); // Caches "I'm listening", "Go ahead", etc.

// Get TTS (cache hit = instant, cache miss = synthesize + cache)
const audio = await getTTS("Hello world");

User Settings Integration

Phase 9 settings are stored in voiceSettingsStore:

File Reference (Phase 9)

Future Work

• Metrics export to backend: Send metrics to backend for aggregation/alerting ✓ Implemented
• Barge-in support: Allow user to interrupt AI responses ✓ Implemented (2025-11-28)
• Audio overlap prevention: Prevent multiple responses playing simultaneously ✓ Implemented (2025-11-28)
• Per-user voice preferences: Backend persistence for TTS settings ✓ Implemented (2025-11-29)
• Context-aware voice styles: Auto-detect tone from content ✓ Implemented (2025-11-29)
• Aggressive latency optimization: 200ms VAD, 256-sample chunks, 300ms reconnect ✓ Implemented (2025-11-29)
• Observability & Monitoring (Phase 3): Error taxonomy, metrics, SLO alerts, telemetry ✓ Implemented (2025-12-02)
• Phase 7: Multilingual Support: Auto language detection, accent profiles, language switch confidence ✓ Implemented (2025-12-03)
• Phase 8: Voice Calibration: Personalized VAD thresholds, calibration wizard, adaptive learning ✓ Implemented (2025-12-03)
• Phase 9: Offline Fallback: Network monitoring, offline VAD, TTS caching, quality-based switching ✓ Implemented (2025-12-03)
• Phase 10: Conversation Intelligence: Sentiment tracking, discourse analysis, response recommendations ✓ Implemented (2025-12-03)

Voice Mode Enhancement - 10 Phase Plan ✅ COMPLETE (2025-12-03)

A comprehensive enhancement transforming voice mode into a human-like conversational partner with medical dictation:

• Phase 1: Emotional Intelligence (Hume AI) ✓ Complete
• Phase 2: Backchanneling System ✓ Complete
• Phase 3: Prosody Analysis ✓ Complete
• Phase 4: Memory & Context System ✓ Complete
• Phase 5: Advanced Turn-Taking ✓ Complete
• Phase 6: Variable Response Timing ✓ Complete
• Phase 7: Conversational Repair ✓ Complete
• Phase 8: Medical Dictation Core ✓ Complete
• Phase 9: Patient Context Integration ✓ Complete
• Phase 10: Frontend Integration & Analytics ✓ Complete

Full documentation: VOICE_MODE_ENHANCEMENT_10_PHASE.md

Remaining Tasks

• Voice→chat transcript content E2E: Test actual transcript content in chat timeline
• Error tracking integration: Send errors to Sentry/similar
• Audio level visualization: Show real-time audio level meter during recording

Voice Mode Settings Guide

This guide explains how to use and configure Voice Mode settings in VoiceAssist.

Overview

Voice Mode provides real-time voice conversations with the AI assistant. Users can customize their voice experience through the settings panel, including voice selection, language preferences, TTS quality parameters, and behavior options.

Voice Mode Overhaul (2025-11-29): Added backend persistence for voice preferences, context-aware voice style detection, and advanced TTS quality controls.

Phase 7-10 Enhancements (2025-12-03): Added multilingual support with auto-detection, voice calibration, offline fallback with network monitoring, and conversation intelligence features.

Accessing Settings

1. Open Voice Mode by clicking the voice button in the chat interface
2. Click the gear icon in the Voice Mode panel header
3. The settings modal will appear

Available Settings

Voice Selection

Choose from 6 different AI voices:

• Alloy - Neutral, balanced voice (default)
• Echo - Warm, friendly voice
• Fable - Expressive, narrative voice
• Onyx - Deep, authoritative voice
• Nova - Energetic, bright voice
• Shimmer - Soft, calming voice

Language

Select your preferred conversation language:

• English (default)
• Spanish
• French
• German
• Italian
• Portuguese

Voice Detection Sensitivity (0-100%)

Controls how sensitive the voice activity detection is:

• Lower values (0-30%): Less sensitive, requires louder/clearer speech
• Medium values (40-60%): Balanced detection (recommended)
• Higher values (70-100%): More sensitive, may pick up background noise

Auto-start Voice Mode

When enabled, Voice Mode will automatically open when you start a new chat or navigate to the chat page. This is useful for voice-first interactions.

Show Status Hints

When enabled, displays helpful tips and instructions in the Voice Mode panel. Disable if you're familiar with the interface and want a cleaner view.

Context-Aware Voice Style (New)

When enabled, the AI automatically adjusts its voice tone based on the content being spoken:

• Calm: Default for medical explanations (stable, measured pace)
• Urgent: For medical warnings/emergencies (dynamic, faster)
• Empathetic: For sensitive health topics (warm, slower)
• Instructional: For step-by-step guidance (clear, deliberate)
• Conversational: For general chat (natural, varied)

The system detects keywords and patterns to select the appropriate style, then blends it with your base preferences (60% your settings, 40% style preset).

Advanced Voice Quality (New)

Expand this section to fine-tune TTS output parameters:

• Voice Stability (0-100%): Lower = more expressive/varied, Higher = more consistent
• Voice Clarity (0-100%): Higher values produce clearer, more consistent voice
• Expressiveness (0-100%): Higher values add more emotion and style variation

These settings primarily affect ElevenLabs TTS but also influence context-aware style blending for OpenAI TTS.

Phase 7: Language & Detection Settings

Auto-Detect Language

When enabled, the system automatically detects the language being spoken and adjusts processing accordingly. This is useful for multilingual users who switch between languages naturally.

• Default: Enabled
• Store Key: autoLanguageDetection

Language Switch Confidence (0-100%)

Controls how confident the system must be before switching to a detected language. Higher values prevent false-positive language switches.

• Lower values (50-70%): More responsive language switching, but may switch accidentally on similar-sounding phrases

• Medium values (70-85%): Balanced detection (recommended)

• Higher values (85-100%): Very confident switching, stays in current language unless clearly different

• Default: 75%

• Store Key: languageSwitchConfidence

Accent Profile

Select a regional accent profile to improve speech recognition accuracy for your specific accent or dialect.

• Default: None (auto-detect)
• Available Profiles: en-us-midwest, en-gb-london, en-au-sydney, ar-eg-cairo, ar-sa-riyadh, etc.
• Store Key: accentProfileId

Phase 8: Voice Calibration Settings

Voice calibration optimizes the VAD (Voice Activity Detection) thresholds specifically for your voice and environment.

Calibration Status

Shows whether voice calibration has been completed:

• Not Calibrated: Default state, using generic thresholds
• Calibrated: Personal thresholds active (shows last calibration date)

Recalibrate Button

Launches the calibration wizard to:

1. Record ambient noise samples
2. Record your speaking voice at different volumes
3. Compute personalized VAD thresholds

Calibration takes approximately 30-60 seconds.

Personalized VAD Threshold

After calibration, the system uses a custom threshold tuned to your voice:

• Store Key: personalizedVadThreshold
• Range: 0.0-1.0 (null if not calibrated)

Adaptive Learning

When enabled, the system continuously learns from your voice patterns and subtly adjusts thresholds over time.

• Default: Enabled
• Store Key: enableBehaviorLearning

Phase 9: Offline Mode Settings

Configure how the voice assistant behaves when network connectivity is poor or unavailable.

Enable Offline Fallback

When enabled, the system automatically switches to offline VAD processing when:

• Network is offline

• Health check fails consecutively

• Network quality drops below threshold

• Default: Enabled

• Store Key: enableOfflineFallback

Prefer Local VAD

Force the use of local (on-device) VAD processing even when network is available. Useful for:

• Privacy-conscious users who don't want audio sent to servers

• Environments with unreliable connectivity

• Lower latency at the cost of accuracy

• Default: Disabled

• Store Key: preferOfflineVAD

TTS Audio Caching

When enabled, previously synthesized audio responses are cached locally for:

• Faster playback of repeated phrases

• Offline playback of cached responses

• Reduced bandwidth and API costs

• Default: Enabled

• Store Key: ttsCacheEnabled

Network Quality Monitoring

The system continuously monitors network quality and categorizes it into five levels:

Network status is displayed in the voice panel header when quality is degraded.

Phase 10: Conversation Intelligence Settings

These settings control advanced AI features that enhance conversation quality.

Enable Sentiment Tracking

When enabled, the AI tracks emotional tone throughout the conversation and adapts its responses accordingly.

• Default: Enabled
• Store Key: enableSentimentTracking

Enable Discourse Analysis

Tracks conversation structure (topic changes, question chains, clarifications) to provide more contextually aware responses.

• Default: Enabled
• Store Key: enableDiscourseAnalysis

Enable Response Recommendations

The AI suggests relevant follow-up questions or actions based on conversation context.

• Default: Enabled
• Store Key: enableResponseRecommendations

Show Suggested Follow-Ups

Display AI-suggested follow-up questions after responses. These appear as clickable chips below the assistant's message.

• Default: Enabled
• Store Key: showSuggestedFollowUps

Privacy Settings

Store Transcript History

When enabled, voice transcripts are stored in the conversation history. Disable for ephemeral voice sessions.

• Default: Enabled
• Store Key: storeTranscriptHistory

Share Anonymous Analytics

Opt-in to share anonymized voice interaction metrics to help improve the service. No transcript content or personal data is shared - only timing metrics (latency, error rates).

• Default: Disabled
• Store Key: shareAnonymousAnalytics

Persistence

Voice preferences are now stored in two locations for maximum reliability:

1. Backend API (Primary): Settings are synced to /api/voice/preferences and stored in the database. This enables cross-device settings sync when logged in.

2. Local Storage (Fallback): Settings are also cached locally under voiceassist-voice-settings for offline access and faster loading.

Changes are debounced (1 second) before being sent to the backend to reduce API calls while editing.

Resetting to Defaults

Click "Reset to defaults" in the settings modal to restore all settings to their original values:

Core Settings

• Voice: Alloy
• Language: English
• VAD Sensitivity: 50%
• Auto-start: Disabled
• Show hints: Enabled
• Context-aware style: Enabled
• Stability: 50%
• Clarity: 75%
• Expressiveness: 0%

Phase 7 Defaults

• Auto Language Detection: Enabled
• Language Switch Confidence: 75%
• Accent Profile ID: null

Phase 8 Defaults

• VAD Calibrated: false
• Last Calibration Date: null
• Personalized VAD Threshold: null
• Adaptive Learning: Enabled

Phase 9 Defaults

• Offline Fallback: Enabled
• Prefer Local VAD: Disabled
• TTS Cache: Enabled

Phase 10 Defaults

• Sentiment Tracking: Enabled
• Discourse Analysis: Enabled
• Response Recommendations: Enabled
• Show Suggested Follow-Ups: Enabled

Privacy Defaults

• Store Transcript History: Enabled
• Share Anonymous Analytics: Disabled

Reset also syncs to the backend via POST /api/voice/preferences/reset.

Voice Preferences API (New)

The following API endpoints manage voice preferences:

Response Headers

TTS synthesis requests now include additional headers:

• X-TTS-Provider: Which provider was used (openai or elevenlabs)
• X-TTS-Fallback: Whether fallback was used (true/false)
• X-TTS-Style: Detected style if context-aware is enabled

Technical Details

Store Location

Settings are managed by a Zustand store with persistence:

apps/web-app/src/stores/voiceSettingsStore.ts

Component Locations

• Settings UI: apps/web-app/src/components/voice/VoiceModeSettings.tsx
• Enhanced Settings: apps/web-app/src/components/voice/VoiceSettingsEnhanced.tsx
• Calibration Dialog: apps/web-app/src/components/voice/CalibrationDialog.tsx

Phase 9 Offline/Network Files

• Network Monitor: apps/web-app/src/lib/offline/networkMonitor.ts
• WebRTC VAD: apps/web-app/src/lib/offline/webrtcVAD.ts
• Offline Types: apps/web-app/src/lib/offline/types.ts
• Network Status Hook: apps/web-app/src/hooks/useNetworkStatus.ts
• Offline VAD Hook: apps/web-app/src/hooks/useOfflineVAD.ts

Backend Files (New)

• Model: services/api-gateway/app/models/user_voice_preferences.py
• Style Detector: services/api-gateway/app/services/voice_style_detector.py
• API Endpoints: services/api-gateway/app/api/voice.py (preferences section)
• Schemas: services/api-gateway/app/api/voice_schemas/schemas.py

Frontend Sync Hook (New)

apps/web-app/src/hooks/useVoicePreferencesSync.ts

Handles loading/saving preferences to backend with debouncing.

Integration Points

• VoiceModePanel.tsx - Displays settings button and uses store values
• MessageInput.tsx - Reads autoStartOnOpen for auto-open behavior
• useVoicePreferencesSync.ts - Backend sync on auth and setting changes

Advanced: Voice Mode Pipeline

Settings are not just UI preferences - they propagate into real-time voice sessions:

• Voice/Language: Sent to /api/voice/realtime-session and used by OpenAI Realtime API
• VAD Sensitivity: Mapped to server-side VAD threshold (0→insensitive, 100→sensitive)

For comprehensive pipeline documentation including backend integration, WebSocket connections, and metrics, see VOICE_MODE_PIPELINE.md.

Development: Running Tests

Run the voice settings test suites individually to avoid memory issues:

cd apps/web-app

# Unit tests for voice settings store (core)
npx vitest run src/stores/__tests__/voiceSettingsStore.test.ts --reporter=dot

# Unit tests for voice settings store (Phase 7-10)
npx vitest run src/stores/__tests__/voiceSettingsStore-phase7-10.test.ts --reporter=dot

# Unit tests for network monitor
npx vitest run src/lib/offline/__tests__/networkMonitor.test.ts --reporter=dot

# Component tests for VoiceModeSettings
npx vitest run src/components/voice/__tests__/VoiceModeSettings.test.tsx --reporter=dot

# Integration tests for MessageInput voice settings
npx vitest run src/components/chat/__tests__/MessageInput-voice-settings.test.tsx --reporter=dot

Test Coverage

The test suites cover:

voiceSettingsStore.test.ts (17 tests)

• Default values verification
• All setter functions (voice, language, sensitivity, toggles)
• VAD sensitivity clamping (0-100 range)
• Reset functionality
• LocalStorage persistence

voiceSettingsStore-phase7-10.test.ts (41 tests)

• Phase 7: Multilingual settings (accent profile, auto-detection, confidence)
• Phase 8: Calibration settings (VAD calibrated, dates, thresholds)
• Phase 9: Offline mode settings (fallback, prefer offline VAD, TTS cache)
• Phase 10: Conversation intelligence (sentiment, discourse, recommendations)
• Privacy settings (transcript history, anonymous analytics)
• Persistence tests for all Phase 7-10 settings
• Reset tests verifying all defaults

networkMonitor.test.ts (13 tests)

• Initial state detection (online/offline)
• Health check latency measurement
• Quality computation from latency thresholds
• Consecutive failure handling before marking unhealthy
• Subscription/unsubscription for status changes
• Custom configuration (latency thresholds, health check URL)
• Offline detection via navigator.onLine

VoiceModeSettings.test.tsx (25 tests)

• Modal visibility (isOpen prop)
• Current settings display
• Settings updates via UI interactions
• Reset with confirmation
• Close behavior (Done, X, backdrop)
• Accessibility (labels, ARIA attributes)

MessageInput-voice-settings.test.tsx (12 tests)

• Auto-open via store setting (autoStartOnOpen)
• Auto-open via prop (autoOpenRealtimeVoice)
• Combined settings behavior
• Voice/language display in panel header
• Status hints visibility toggle

Total: 108+ tests for voice settings and related functionality.

Notes

• Tests mock useRealtimeVoiceSession and WaveformVisualizer to avoid browser API dependencies
• Run tests individually rather than the full suite to prevent memory issues
• All tests use Vitest + React Testing Library
• Phase 7-10 tests also mock fetch and performance.now for network monitoring

Thinker Service

Location: services/api-gateway/app/services/thinker_service.py Status: Production Ready Last Updated: 2025-12-01

Overview

The ThinkerService is the reasoning engine of the Thinker-Talker voice pipeline. It manages conversation context, orchestrates LLM interactions, and handles tool calling with result injection.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      ThinkerService                              │
│                                                                  │
│  ┌──────────────────┐    ┌──────────────────┐                   │
│  │ ConversationContext │◄──│   ThinkerSession  │                 │
│  │ (shared memory)     │    │   (per-request)   │                │
│  └──────────────────┘    └──────────────────┘                   │
│           │                        │                             │
│           │                        ▼                             │
│           │              ┌──────────────────┐                   │
│           │              │    LLMClient     │                   │
│           │              │   (GPT-4o)       │                   │
│           │              └──────────────────┘                   │
│           │                        │                             │
│           │                        ▼                             │
│           │              ┌──────────────────┐                   │
│           │              │   ToolRegistry   │                   │
│           │              │ (calendar, search,│                  │
│           │              │  medical, KB)     │                   │
│           └──────────────┴──────────────────┘                   │
└─────────────────────────────────────────────────────────────────┘

Classes

ThinkerService

Main service class (singleton pattern).

from app.services.thinker_service import thinker_service

# Create a session for a conversation
session = thinker_service.create_session(
    conversation_id="conv-123",
    on_token=handle_token,         # Called for each LLM token
    on_tool_call=handle_tool_call, # Called when tool is invoked
    on_tool_result=handle_result,  # Called when tool returns
    user_id="user-456",            # Required for authenticated tools
)

# Process user input
response = await session.think("What's on my calendar today?")

Methods

ThinkerSession

Session class for processing individual requests.

class ThinkerSession:
    """
    A single thinking session with streaming support.

    Manages the flow:
    1. Receive user input
    2. Add to conversation context
    3. Call LLM with streaming
    4. Handle tool calls if needed
    5. Stream response tokens to callback
    """

Methods

Properties

ConversationContext

Manages conversation history with smart trimming.

class ConversationContext:
    MAX_HISTORY_MESSAGES = 20    # Maximum messages to retain
    MAX_CONTEXT_TOKENS = 8000    # Token budget for context

    def __init__(self, conversation_id: str, system_prompt: str = None):
        self.conversation_id = conversation_id
        self.messages: List[ConversationMessage] = []
        self.system_prompt = system_prompt or self._default_system_prompt()

Smart Trimming

When message count exceeds MAX_HISTORY_MESSAGES, the context performs smart trimming:

def _smart_trim(self) -> None:
    """
    Trim messages while preserving tool call chains.

    OpenAI requires: assistant (with tool_calls) -> tool (with tool_call_id)
    We can't break this chain or the API will reject the request.
    """

Rules:

• Never trim an assistant message if the next message is a tool result
• Never trim a tool message (it needs its preceding assistant message)
• Find the first safe trim point that doesn't break chains

Methods

ToolRegistry

Registry for available tools.

class ToolRegistry:
    def register(
        self,
        name: str,
        description: str,
        parameters: Dict,
        handler: Callable[[Dict], Awaitable[Any]],
    ) -> None:
        """Register a tool with its schema and handler."""

    def get_tools_schema(self) -> List[Dict]:
        """Get all tool schemas for LLM API."""

    async def execute(self, tool_name: str, arguments: Dict, user_id: str) -> Any:
        """Execute a tool and return its result."""

Data Classes

ThinkingState

class ThinkingState(str, Enum):
    IDLE = "idle"           # Waiting for input
    PROCESSING = "processing"  # Building request
    TOOL_CALLING = "tool_calling"  # Executing tool
    GENERATING = "generating"  # Streaming response
    COMPLETE = "complete"    # Finished successfully
    CANCELLED = "cancelled"  # User interrupted
    ERROR = "error"          # Error occurred

ConversationMessage

@dataclass
class ConversationMessage:
    role: str              # "user", "assistant", "system", "tool"
    content: str
    message_id: str        # Auto-generated UUID
    timestamp: float       # Unix timestamp
    source_mode: str       # "chat" or "voice"
    tool_call_id: str      # For tool results
    tool_calls: List[Dict] # For assistant messages with tool calls
    citations: List[Dict]  # Source citations

ThinkerResponse

@dataclass
class ThinkerResponse:
    text: str                    # Complete response text
    message_id: str              # Unique ID
    citations: List[Dict]        # Source citations
    tool_calls_made: List[str]   # Names of tools called
    latency_ms: int              # Total processing time
    tokens_used: int             # Token count
    state: ThinkingState         # Final state

ThinkerMetrics

@dataclass
class ThinkerMetrics:
    total_tokens: int = 0
    tool_calls_count: int = 0
    first_token_latency_ms: int = 0
    total_latency_ms: int = 0
    cancelled: bool = False

Available Tools

The ThinkerService automatically registers tools from the unified ToolService:

System Prompt

The default system prompt includes:

1. Current Time Context: Dynamic date/time with relative calculations
2. Conversation Memory: Instructions to use conversation history
3. Tool Usage Guidelines: When and how to use each tool
4. Response Style: Concise, natural, voice-optimized

def _default_system_prompt(self) -> str:
    tz = pytz.timezone("America/New_York")
    now = datetime.now(tz)

    return f"""You are VoiceAssist, a helpful AI voice assistant.

CURRENT TIME CONTEXT:
- Current date: {now.strftime("%A, %B %d, %Y")}
- Current time: {now.strftime("%I:%M %p %Z")}

CONVERSATION MEMORY:
You have access to the full conversation history...

AVAILABLE TOOLS:
- calendar_create_event: Create events...
- web_search: Search the web...
...

KEY BEHAVIORS:
- Keep responses concise and natural for voice
- Use short sentences (max 15-20 words)
- Avoid abbreviations - say "blood pressure" not "BP"
"""

Usage Examples

Basic Query Processing

from app.services.thinker_service import thinker_service

async def handle_voice_query(conversation_id: str, transcript: str, user_id: str):
    # Token streaming callback
    async def on_token(token: str):
        await send_to_tts(token)

    # Create session with callbacks
    session = thinker_service.create_session(
        conversation_id=conversation_id,
        on_token=on_token,
        user_id=user_id,
    )

    # Process the transcript
    response = await session.think(transcript, source_mode="voice")

    print(f"Response: {response.text}")
    print(f"Tools used: {response.tool_calls_made}")
    print(f"Latency: {response.latency_ms}ms")

With Tool Call Handling

async def handle_tool_call(event: ToolCallEvent):
    """Called when LLM decides to call a tool."""
    await send_to_client({
        "type": "tool.call",
        "tool_name": event.tool_name,
        "arguments": event.arguments,
    })

async def handle_tool_result(event: ToolResultEvent):
    """Called when tool execution completes."""
    await send_to_client({
        "type": "tool.result",
        "tool_name": event.tool_name,
        "result": event.result,
    })

session = thinker_service.create_session(
    conversation_id="conv-123",
    on_token=on_token,
    on_tool_call=handle_tool_call,
    on_tool_result=handle_tool_result,
    user_id="user-456",
)

Cancellation (Barge-in)

# Store session reference
active_session = thinker_service.create_session(...)

# When user barges in:
async def handle_barge_in():
    await active_session.cancel()
    print(f"Cancelled: {active_session.is_cancelled()}")

Context Persistence

Conversation contexts are persisted across turns:

# Class-level storage
_conversation_contexts: Dict[str, ConversationContext] = {}
_context_last_access: Dict[str, float] = {}
CONTEXT_TTL_SECONDS = 3600  # 1 hour TTL

• Contexts are automatically cleaned up after 1 hour of inactivity
• Same conversation_id reuses existing context
• Context persists across voice and chat modes

Error Handling

try:
    response = await session.think(transcript)
except Exception as e:
    # Errors are caught and returned in response
    response = ThinkerResponse(
        text=f"I apologize, but I encountered an error: {str(e)}",
        message_id=message_id,
        state=ThinkingState.ERROR,
    )

Related Documentation

• Thinker-Talker Pipeline Overview
• Talker Service
• Tool Service

Talker Service

Location: services/api-gateway/app/services/talker_service.py Status: Production Ready Last Updated: 2025-12-01

Overview

The TalkerService handles text-to-speech synthesis for the Thinker-Talker voice pipeline. It streams LLM tokens through a sentence chunker and synthesizes speech via ElevenLabs for gapless audio playback.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                       TalkerService                              │
│                                                                  │
│   LLM Tokens ──►┌──────────────────┐                            │
│                 │ Markdown Buffer  │  (accumulates for pattern  │
│                 │                  │   detection before strip)   │
│                 └────────┬─────────┘                            │
│                          │                                       │
│                          ▼                                       │
│                 ┌──────────────────┐                            │
│                 │ SentenceChunker  │  (splits at natural        │
│                 │ (40-120-200 chars)│   boundaries)              │
│                 └────────┬─────────┘                            │
│                          │                                       │
│                          ▼                                       │
│                 ┌──────────────────┐                            │
│                 │ strip_markdown   │  (removes **bold**,        │
│                 │ _for_tts()       │   [links](url), LaTeX)     │
│                 └────────┬─────────┘                            │
│                          │                                       │
│                          ▼                                       │
│                 ┌──────────────────┐                            │
│                 │ ElevenLabs TTS   │  (streaming synthesis      │
│                 │ (sequential)     │   with previous_text)      │
│                 └────────┬─────────┘                            │
│                          │                                       │
│                          ▼                                       │
│                   Audio Chunks ──► on_audio_chunk callback       │
└─────────────────────────────────────────────────────────────────┘

Classes

TalkerService

Main service class (singleton pattern).

from app.services.talker_service import talker_service

# Check if TTS is available
if talker_service.is_enabled():
    # Start a speaking session (uses DEFAULT_VOICE_ID from voice_constants.py)
    session = await talker_service.start_session(
        on_audio_chunk=handle_audio,
        voice_config=VoiceConfig(
            # voice_id defaults to DEFAULT_VOICE_ID (Brian)
            stability=0.65,
        ),
    )

    # Feed tokens from LLM
    for token in llm_stream:
        await session.add_token(token)

    # Finish and get metrics
    metrics = await session.finish()

Methods

TalkerSession

Session class for streaming TTS.

class TalkerSession:
    """
    A single TTS speaking session with streaming support.

    Manages the flow:
    1. Receive LLM tokens
    2. Chunk into sentences
    3. Synthesize each sentence
    4. Stream audio chunks to callback
    """

Methods

Properties

AudioQueue

Queue management for gapless playback.

class AudioQueue:
    """
    Manages audio chunks for gapless playback with cancellation support.

    Features:
    - Async queue for audio chunks
    - Cancellation clears pending audio
    - Tracks queue state
    """

    async def put(self, chunk: AudioChunk) -> bool
    async def get(self) -> Optional[AudioChunk]
    async def cancel(self) -> None
    def finish(self) -> None
    def reset(self) -> None

Data Classes

TalkerState

class TalkerState(str, Enum):
    IDLE = "idle"           # Ready for input
    SPEAKING = "speaking"   # Synthesizing/playing
    CANCELLED = "cancelled" # Interrupted by barge-in

TTSProvider

class TTSProvider(str, Enum):
    ELEVENLABS = "elevenlabs"
    OPENAI = "openai"  # Fallback

VoiceConfig

Note: Default voice is configured in app/core/voice_constants.py. See Voice Configuration for details.

from app.core.voice_constants import DEFAULT_VOICE_ID, DEFAULT_TTS_MODEL

@dataclass
class VoiceConfig:
    provider: TTSProvider = TTSProvider.ELEVENLABS
    voice_id: str = DEFAULT_VOICE_ID  # Brian (from voice_constants.py)
    model_id: str = DEFAULT_TTS_MODEL  # eleven_flash_v2_5
    stability: float = 0.65         # 0.0-1.0, higher = consistent
    similarity_boost: float = 0.80  # 0.0-1.0, higher = clearer
    style: float = 0.15             # 0.0-1.0, lower = natural
    use_speaker_boost: bool = True
    output_format: str = "pcm_24000"

AudioChunk

@dataclass
class AudioChunk:
    data: bytes          # Raw audio bytes
    format: str          # "pcm16" or "mp3"
    is_final: bool       # True for last chunk
    sentence_index: int  # Which sentence this is from
    latency_ms: int      # Time since synthesis started

TalkerMetrics

@dataclass
class TalkerMetrics:
    sentences_processed: int = 0
    total_chars_synthesized: int = 0
    total_audio_bytes: int = 0
    total_latency_ms: int = 0
    first_audio_latency_ms: int = 0
    cancelled: bool = False

Sentence Chunking

The TalkerSession uses SentenceChunker with these settings:

self._chunker = SentenceChunker(
    ChunkerConfig(
        min_chunk_chars=40,    # Avoid tiny fragments
        optimal_chunk_chars=120,  # Full sentences
        max_chunk_chars=200,   # Allow complete thoughts
    )
)

Why These Settings?

Trade-off: Larger chunks = better prosody but higher latency to first audio.

Markdown Stripping

LLM responses often contain markdown that sounds unnatural when spoken:

def strip_markdown_for_tts(text: str) -> str:
    """
    Converts:
    - [Link Text](URL) → "Link Text"
    - **bold** → "bold"
    - *italic* → "italic"
    - `code` → "code"
    - ```blocks``` → (removed)
    - # Headers → "Headers"
    - LaTeX formulas → (removed)
    """

Markdown-Aware Token Buffering

The TalkerSession buffers tokens to detect incomplete patterns:

def _process_markdown_token(self, token: str) -> str:
    """
    Accumulates tokens to detect patterns that should be stripped:
    - Markdown links: [text](url) - wait for closing )
    - LaTeX display: [ ... ] with backslashes
    - LaTeX inline: \\( ... \\)
    - Bold/italic: **text** - wait for closing **
    """

This prevents sending "[Link Te" to TTS before we know it's a markdown link.

Voice Continuity

For consistent voice across sentences:

async for audio_data in self._elevenlabs.synthesize_stream(
    text=tts_text,
    previous_text=self._previous_text,  # Context for voice continuity
    ...
):
    ...

# Update for next synthesis
self._previous_text = tts_text

The previous_text parameter helps ElevenLabs maintain consistent prosody.

Sequential Synthesis

To prevent voice variations between chunks:

# Semaphore ensures one synthesis at a time
self._synthesis_semaphore = asyncio.Semaphore(1)

async with self._synthesis_semaphore:
    async for audio_data in self._elevenlabs.synthesize_stream(...):
        ...

Parallel synthesis can cause noticeable voice quality differences between sentences.

Usage Examples

Basic Token Streaming

async def handle_llm_response(llm_stream):
    async def on_audio_chunk(chunk: AudioChunk):
        # Send to client via WebSocket
        await websocket.send_json({
            "type": "audio.output",
            "audio": base64.b64encode(chunk.data).decode(),
            "is_final": chunk.is_final,
        })

    session = await talker_service.start_session(on_audio_chunk=on_audio_chunk)

    async for token in llm_stream:
        await session.add_token(token)

    metrics = await session.finish()
    print(f"Synthesized {metrics.sentences_processed} sentences")
    print(f"First audio in {metrics.first_audio_latency_ms}ms")

Custom Voice Configuration

config = VoiceConfig(
    voice_id="21m00Tcm4TlvDq8ikWAM",  # Rachel (female)
    model_id="eleven_flash_v2_5",      # Lower latency
    stability=0.65,                    # More variation
    similarity_boost=0.90,             # Very clear
    style=0.15,                        # Slightly expressive
)

session = await talker_service.start_session(
    on_audio_chunk=handle_audio,
    voice_config=config,
)

Handling Barge-in

active_session = None

async def start_speaking(llm_stream):
    global active_session
    active_session = await talker_service.start_session(on_audio_chunk=send_audio)

    for token in llm_stream:
        if active_session.is_cancelled():
            break
        await active_session.add_token(token)

    await active_session.finish()

async def handle_barge_in():
    global active_session
    if active_session:
        await active_session.cancel()
        # Cancels pending synthesis and clears audio queue

Simple Text Synthesis

# For non-streaming use cases
async for audio_chunk in talker_service.synthesize_text(
    text="Hello, how can I help you today?",
    voice_config=VoiceConfig(voice_id="TxGEqnHWrfWFTfGW9XjX"),
):
    await send_audio(audio_chunk)

Available Voices

voices = talker_service.get_available_voices()
# Returns:
[
    {"id": "TxGEqnHWrfWFTfGW9XjX", "name": "Josh", "gender": "male", "premium": True},
    {"id": "pNInz6obpgDQGcFmaJgB", "name": "Adam", "gender": "male", "premium": True},
    {"id": "EXAVITQu4vr4xnSDxMaL", "name": "Bella", "gender": "female", "premium": True},
    {"id": "21m00Tcm4TlvDq8ikWAM", "name": "Rachel", "gender": "female", "premium": True},
    # ... more voices
]

Performance Tuning

Latency Optimization

Quality Optimization

Error Handling

Synthesis errors don't fail the entire session:

async def _synthesize_sentence(self, sentence: str) -> None:
    try:
        async for audio_data in self._elevenlabs.synthesize_stream(...):
            if self._state == TalkerState.CANCELLED:
                return
            await self._on_audio_chunk(chunk)
    except Exception as e:
        logger.error(f"TTS synthesis error: {e}")
        # Session continues, just skips this sentence

Related Documentation

• Thinker-Talker Pipeline Overview
• Thinker Service
• Voice Pipeline WebSocket API

Voice Pipeline WebSocket API

Endpoint: wss://{host}/api/voice/pipeline-ws Protocol: JSON over WebSocket Status: Production Ready Last Updated: 2025-12-02

Overview

The Voice Pipeline WebSocket provides bidirectional communication for the Thinker-Talker voice mode. It handles audio streaming, transcription, LLM responses, and TTS playback.

Connection

Authentication

Include JWT token in connection URL or headers:

const ws = new WebSocket(`wss://assist.asimo.io/api/voice/pipeline-ws?token=${accessToken}`);

Connection Lifecycle

1. Client connects with auth token
   │
2. Server accepts, creates pipeline session
   │
3. Server sends: session.ready
   │
4. Client sends: session.init (optional config)
   │
5. Server acknowledges: session.init.ack
   │
6. Voice mode active - bidirectional streaming
   │
7. Client or server closes connection

Message Format

All messages are JSON objects with a type field:

{
  "type": "message_type",
  "field1": "value1",
  "field2": "value2"
}

Client → Server Messages

session.init

Initialize or reconfigure the session.

{
  "type": "session.init",
  "conversation_id": "conv-123",
  "voice_settings": {
    "voice_id": "TxGEqnHWrfWFTfGW9XjX",
    "language": "en",
    "barge_in_enabled": true
  }
}