Voice Mode Architecture

Overview

VoiceAssist implements a sophisticated voice-first interface for healthcare professionals, enabling natural spoken interactions with an AI medical assistant. The system uses a Thinker/Talker pipeline architecture that decouples speech recognition, language model reasoning, and speech synthesis for maximum flexibility and low latency.

High-Level Architecture

Current Implementation of Voice Mode

End-to-End Pipeline

The voice interaction follows this sequence:

Audio Capture

Speech-to-Text (STT) Providers

LLM / Assistant Layer

The Thinker Service (thinker_service.py) handles language model reasoning with intelligent routing:

# Query classification determines urgency
URGENT → prioritized, faster response
SIMPLE → direct answer, minimal context
COMPLEX → multi-hop reasoning, RAG retrieval
CLARIFICATION → follow-up questions

Text-to-Speech (TTS) Providers

Streaming and Latency Behavior

Streaming Architecture

All pipeline components support streaming to minimize perceived latency:

Latency Targets

Voice Quality Presets

Users can select latency vs. quality trade-offs:

// voiceSettingsStore.ts
type VoiceQualityPreset = 'speed' | 'balanced' | 'natural';
 
const presets = {
  speed: { ttfa: '100-150ms', description: 'Fastest response' },
  balanced: { ttfa: '200-250ms', description: 'Recommended default' },
  natural: { ttfa: '300-400ms', description: 'Most natural prosody' }
};

VAD and End-of-Utterance Detection

The system determines when the user has finished speaking using:

1. Deepgram VAD Events: Server-side voice activity detection
2. Silence Threshold: 800ms of silence triggers end-of-utterance
3. VAD Sensitivity: 200ms minimum speech duration to avoid false triggers

Barge-In Support

Users can interrupt the AI's response mid-playback:

• Detection: barge_in_classifier.py monitors for new speech during playback
• Action: Current audio playback stops, new utterance is processed
• UI: VoiceBargeInIndicator.tsx provides visual feedback

Multilingual and Pronunciation Behavior

Supported Languages

Mixed-Language Support

Workarounds:

• Configure STT for the dominant language
• Use medical terminology in the configured language
• Rely on TTS's multilingual model for pronunciation

Pronunciation Handling

Known Issues:

• Uncommon drug names may be mispronounced
• Eponyms (e.g., "Parkinson's", "Alzheimer's") generally work well
• Abbreviations (e.g., "mg", "mL") require SSML hints

Architecture and Module Integration

Backend Service Structure

The voice pipeline is implemented across multiple services in services/api-gateway/app/services/:

services/
├── voice_pipeline_service.py      # Main orchestrator
├── streaming_stt_service.py       # Deepgram/Whisper STT
├── thinker_service.py             # LLM reasoning
├── talker_service.py              # TTS orchestration
├── voice_websocket_handler.py     # WebSocket management
├── thinker_talker_websocket_handler.py  # T/T protocol
├── voice_activity_detector.py     # VAD logic
├── barge_in_classifier.py         # Interrupt detection
├── elevenlabs_service.py          # ElevenLabs client
├── openai_tts_service.py          # OpenAI TTS client
├── ssml_processor.py              # SSML generation
├── emotion_detection_service.py   # User emotion analysis
├── prosody_analysis_service.py    # Speech prosody
├── backchannel_service.py         # Conversational cues
└── dictation_service.py           # Medical dictation

Frontend Hook Structure

Voice features are exposed via React hooks in apps/web-app/src/hooks/:

// Primary hooks (current production)
useThinkerTalkerSession.ts      // Session management
useThinkerTalkerVoiceMode.ts    // Combined session + playback
useTTAudioPlayback.ts           // Audio streaming playback
 
// Supporting hooks
useVoiceMetrics.ts              // Latency tracking
useVoiceModeStateMachine.ts     // State management
useStreamingAudio.ts            // Audio stream handling
useBackchannelAudio.ts          // AI conversational cues
useVoicePreferencesSync.ts      // Settings persistence
 
// Legacy (deprecated)
useRealtimeVoiceSession.ts      // OpenAI Realtime API (deprecated)

Pipeline Modes

The voice pipeline supports multiple operating modes:

Error Handling and Retries

// Circuit breaker pattern for external APIs
const circuitBreaker = {
  failureThreshold: 5,
  recoveryTimeout: 30000, // 30 seconds
  halfOpenRequests: 3
};
 
// Retry strategy
const retryPolicy = {
  maxRetries: 3,
  baseDelay: 1000,
  maxDelay: 10000,
  backoffMultiplier: 2
};

Medical Intelligence and Data Sources

Currently Integrated Sources

RAG Architecture

The system uses Retrieval-Augmented Generation for evidence-based responses:

Medical Embedding Models

Multiple embedding models are available for semantic search:

FHIR Integration

Known Gaps and TODOs

Voice Pipeline Gaps

Medical Intelligence Gaps

Documentation Gaps

Configuration Reference

Environment Variables

# STT Configuration
DEEPGRAM_API_KEY=your-deepgram-key
VOICE_PIPELINE_STT_PRIMARY=deepgram
VOICE_PIPELINE_STT_FALLBACK=whisper
 
# TTS Configuration
ELEVENLABS_API_KEY=your-elevenlabs-key
VOICE_PIPELINE_TTS_PROVIDER=elevenlabs
TTS_VOICE=default-voice-id
 
# LLM Configuration
OPENAI_API_KEY=your-openai-key
LOCAL_LLM_ENDPOINT=http://localhost:11434
 
# Voice Pipeline
VOICE_WS_MAX_INFLIGHT=10
VAD_SILENCE_THRESHOLD_MS=800
VAD_SENSITIVITY_MS=200

User Preferences (voiceSettingsStore)

interface VoiceSettings {
  voiceId: string;           // ElevenLabs voice ID
  language: string;          // ISO language code
  playbackSpeed: number;     // 0.5-2.0x
  stability: number;         // 0.0-1.0
  clarity: number;           // 0.0-1.0
  expressiveness: number;    // 0.0-1.0
  qualityPreset: 'speed' | 'balanced' | 'natural';
  pushToTalk: boolean;
  autoPlay: boolean;
}

Cost Philosophy

Guiding Principles:

1. Quality First: Premium providers (ElevenLabs, Deepgram) are preferred for their superior quality
2. Smart Fallbacks: Cost-effective alternatives only activate when primary providers fail
3. No Downgrades: Never propose replacing current components with cheaper, lower-quality alternatives
4. Measured Upgrades: New features should justify their cost with measurable UX improvements

References

Backend Files

• services/api-gateway/app/services/voice_pipeline_service.py
• services/api-gateway/app/services/streaming_stt_service.py
• services/api-gateway/app/services/thinker_service.py
• services/api-gateway/app/services/talker_service.py
• services/api-gateway/app/services/elevenlabs_service.py

Frontend Files

• apps/web-app/src/hooks/useThinkerTalkerSession.ts
• apps/web-app/src/hooks/useThinkerTalkerVoiceMode.ts
• apps/web-app/src/components/voice/ThinkerTalkerVoicePanel.tsx
• apps/web-app/src/stores/voiceSettingsStore.ts

Related Documentation

• Realtime WebSocket Protocol
• Deployment Guide