World-Class Voice Barge-In Implementation Plan

Goal: Transform VoiceAssist's voice mode from basic interruption handling to a human-like conversational experience with <30ms speech detection, intelligent context-aware interruption handling, natural turn-taking, multilingual support, and adaptive personalization.

Created: 2025-12-02 Revised: 2025-12-04 Status: ✅ Implementation Complete (Phases 1-10)

Table of Contents

1. Executive Summary
2. Current State Analysis
3. Intelligent Barge-In State Machine
4. Phase 1: Neural VAD Integration
5. Phase 2: Instant Response & Feedback
6. Phase 3: Context-Aware Interruption Intelligence
7. Phase 4: Advanced Audio Processing
8. Phase 5: Natural Turn-Taking
9. Phase 6: Full Duplex Experience
10. Phase 7: Multilingual & Accent Support
11. Phase 8: Adaptive Personalization
12. Phase 9: Offline & Low-Latency Fallback
13. Phase 10: Advanced Conversation Management
14. Privacy & Security
15. Continuous Learning Pipeline
16. Testing Strategy
17. Success Metrics
18. File Summary
19. Implementation Timeline

Executive Summary

This plan transforms VoiceAssist's voice mode into a world-class conversational experience that feels like talking to a human. Key innovations include:

Key Targets

Current State Analysis

What Exists Today

• Basic barge-in via response.cancel signal
• Energy-based VAD (simple RMS threshold)
• 300-500ms end-to-end latency
• AudioWorklet with 10.7ms chunks
• Manual barge-in button + auto-detection

Key Gaps for Human-Like Conversation

1. Detection latency: ~50-100ms delay before speech is recognized
2. No immediate feedback: User doesn't know they were "heard" instantly
3. Abrupt cutoff: AI audio stops abruptly (unnatural)
4. No context awareness: System doesn't understand why user interrupted
5. Echo confusion: Sometimes confuses AI audio for user speech
6. Single mode: No distinction between "I want to interject" vs "background noise"
7. English-only: No multilingual backchannel or phrase detection
8. No personalization: One-size-fits-all thresholds
9. Network-dependent: No offline fallback for barge-in detection
10. Tool-call blindness: No safe interruption during external API calls

Current Architecture

User Microphone (16kHz PCM)
       ↓
Deepgram Streaming STT (with Whisper fallback)
       ↓
GPT-4o Thinker (with tool calling support)
       ↓
ElevenLabs Streaming TTS (24kHz PCM)
       ↓
Web Audio API Playback

Intelligent Barge-In State Machine

State Machine Architecture

┌─────────────────────────────────────────────────────────────────────────────────┐
│                    INTELLIGENT BARGE-IN STATE MACHINE                           │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│  ┌─────────┐                                                                    │
│  │  IDLE   │◄──────────────────────────────────────────────────────────────┐   │
│  └────┬────┘                                                               │   │
│       │ connect()                                                          │   │
│       ▼                                                                    │   │
│  ┌──────────────┐                                                          │   │
│  │ CALIBRATING  │──── calibration_complete ────────────────────────┐       │   │
│  │ (noise floor)│                                                  │       │   │
│  └──────┬───────┘                                                  │       │   │
│         │ skip_calibration                                         │       │   │
│         ▼                                                          │       │   │
│  ┌──────────────┐                                                  │       │   │
│  │ CONNECTING   │──────── error ──────────────────────────────────┐│       │   │
│  └──────┬───────┘                                                 ││       │   │
│         │ session.ready                                           ││       │   │
│         ▼                                                         ▼│       │   │
│  ┌──────────────┐                                                  │       │   │
│  │  LISTENING   │◄─────────────────────────────────────────────┐   │       │   │
│  │  (ready)     │                                              │   │       │   │
│  └──────┬───────┘                                              │   │       │   │
│         │ vad.speech_onset (confidence > adaptive_threshold)   │   │       │   │
│         ▼                                                      │   │       │   │
│  ┌──────────────────┐                                          │   │       │   │
│  │ SPEECH_DETECTED  │ ◄── 20-30ms window                       │   │       │   │
│  │ (pre-confirm)    │     for onset detection                  │   │       │   │
│  └──────┬───────────┘                                          │   │       │   │
│         │                                                      │   │       │   │
│         ├─── speech < 100ms + low confidence ───► LISTENING    │   │       │   │
│         │    (false positive / noise)              (cancel)    │   │       │   │
│         │                                                      │   │       │   │
│         │ speech >= 100ms OR high confidence (>0.85)           │   │       │   │
│         ▼                                                      │   │       │   │
│  ┌──────────────────┐                                          │   │       │   │
│  │ USER_SPEAKING    │                                          │   │       │   │
│  │ (confirmed)      │                                          │   │       │   │
│  └──────┬───────────┘                                          │   │       │   │
│         │ silence > adaptive_threshold (200-800ms)             │   │       │   │
│         ▼                                                      │   │       │   │
│  ┌──────────────────┐                                          │   │       │   │
│  │ PROCESSING_STT   │                                          │   │       │   │
│  │ (finalizing)     │                                          │   │       │   │
│  └──────┬───────────┘                                          │   │       │   │
│         │ transcript.complete                                  │   │       │   │
│         ▼                                                      │   │       │   │
│  ┌──────────────────┐                                          │   │       │   │
│  │ PROCESSING_LLM   │─────────────────────────────────────────────────────────┐│
│  │ (thinking/tools) │  ◄── tool_call_in_progress               │   │       │ ││
│  └──────┬───────────┘                                          │   │       │ ││
│         │ response.delta (first token)                         │   │       │ ││
│         ▼                                                      │   │       │ ││
│  ┌──────────────────┐      vad.speech_onset                    │   │       │ ││
│  │ AI_RESPONDING    │◄────────────────────────────┐            │   │       │ ││
│  │ (streaming text) │                             │            │   │       │ ││
│  └──────┬───────────┘                             │            │   │       │ ││
│         │ audio.output (first chunk)              │            │   │       │ ││
│         ▼                                         │            │   │       │ ││
│  ┌──────────────────┐                             │            │   │       │ ││
│  │  AI_SPEAKING     │─────────────────────────────┤            │   │       │ ││
│  │  (playing TTS)   │     (BARGE-IN ZONE)         │            │   │       │ ││
│  └──────┬───────────┘                             │            │   │       │ ││
│         │                                         │            │   │       │ ││
│         │ vad.speech_onset ────────────────────►  │            │   │       │ ││
│         │                                         │            │   │       │ ││
│         │         ┌───────────────────────────────┴──────┐     │   │       │ ││
│         │         │      BARGE-IN CLASSIFICATION         │     │   │       │ ││
│         │         │      (language-aware)                │     │   │       │ ││
│         │         │                                      │     │   │       │ ││
│         │         │  ┌─────────────┐  ┌──────────────┐   │     │   │       │ ││
│         │         │  │BACKCHANNEL  │  │ SOFT_BARGE   │   │     │   │       │ ││
│         │         │  │"uh huh"     │  │ "wait"       │   │     │   │       │ ││
│         │         │  │"yeah" (EN)  │  │ "hold on"    │   │     │   │       │ ││
│         │         │  │"نعم" (AR)   │  │ "actually"   │   │     │   │       │ ││
│         │         │  │"oui" (FR)   │  │ short phrase │   │     │   │       │ ││
│         │         │  └──────┬──────┘  └──────┬───────┘   │     │   │       │ ││
│         │         │         │                │           │     │   │       │ ││
│         │         │         ▼                ▼           │     │   │       │ ││
│         │         │  ┌─────────────┐  ┌──────────────┐   │     │   │       │ ││
│         │         │  │ Continue    │  │ Fade to 20%  │   │     │   │       │ ││
│         │         │  │ AI audio    │  │ Pause LLM    │   │     │   │       │ ││
│         │         │  │ (no action) │  │ Wait 2s      │   │     │   │       │ ││
│         │         │  └─────────────┘  └──────────────┘   │     │   │       │ ││
│         │         │                                      │     │   │       │ ││
│         │         │  ┌──────────────────────────────┐    │     │   │       │ ││
│         │         │  │      HARD_BARGE_IN           │    │     │   │       │ ││
│         │         │  │  Full sentence / question    │    │     │   │       │ ││
│         │         │  │  High confidence speech      │    │     │   │       │ ││
│         │         │  │  Duration > 300ms            │    │     │   │       │ ││
│         │         │  └──────────────┬───────────────┘    │     │   │       │ ││
│         │         │                 │                    │     │   │       │ ││
│         │         │                 ▼                    │     │   │       │ ││
│         │         │  ┌──────────────────────────────┐    │     │   │       │ ││
│         │         │  │ 1. Immediate audio fade (30ms)│   │     │   │       │ ││
│         │         │  │ 2. Check tool-call state     │────┼─────┼───┼───────┼─┘│
│         │         │  │ 3. Safe interrupt/rollback   │    │     │   │       │  │
│         │         │  │ 4. Store interrupted context │    │     │   │       │  │
│         │         │  │ 5. Generate context summary  │    │     │   │       │  │
│         │         │  │ 6. Show visual confirmation  │    │     │   │       │  │
│         │         │  └──────────────────────────────┘    │     │   │       │  │
│         │         └──────────────────────────────────────┘     │   │       │  │
│         │                                                      │   │       │  │
│         │ audio.complete (natural end)                         │   │       │  │
│         └──────────────────────────────────────────────────────┘   │       │  │
│                                                                     │       │  │
│  ┌─────────┐                                                        │       │  │
│  │  ERROR  │◄───────────────────────────────────────────────────────┘       │  │
│  └────┬────┘                                                                 │  │
│       │ retry() or disconnect()                                              │  │
│       └──────────────────────────────────────────────────────────────────────┘  │
│                                                                                  │
│  ┌─────────────────────────────────────────────────────────────────────────┐    │
│  │                    TOOL-CALL INTERRUPT HANDLER                          │    │
│  │  If barge-in during PROCESSING_LLM with active tool call:               │    │
│  │  1. Check tool interruptibility (safe_to_interrupt flag)                │    │
│  │  2. If interruptible: cancel & rollback                                 │    │
│  │  3. If not interruptible: queue barge-in, notify user                   │    │
│  │  4. Log interruption for telemetry                                      │    │
│  └─────────────────────────────────────────────────────────────────────────┘    │
│                                                                                  │
└──────────────────────────────────────────────────────────────────────────────────┘

State Definitions

// New file: apps/web-app/src/hooks/useIntelligentBargeIn/types.ts

export type BargeInState =
  | "idle" // Voice mode inactive
  | "calibrating" // Measuring ambient noise for thresholds
  | "connecting" // Establishing WebSocket
  | "listening" // Ready, waiting for user speech
  | "speech_detected" // VAD triggered, confirming (20-30ms)
  | "user_speaking" // Confirmed user speech
  | "processing_stt" // Finalizing transcript
  | "processing_llm" // LLM generating response (may include tool calls)
  | "ai_responding" // LLM streaming tokens (no audio yet)
  | "ai_speaking" // TTS audio playing
  | "barge_in_detected" // User spoke during AI, classifying
  | "soft_barge" // Soft interruption (AI paused)
  | "awaiting_continuation" // After soft barge, waiting for user
  | "tool_call_pending" // Barge-in queued during non-interruptible tool call
  | "error"; // Error state

export type BargeInClassification =
  | "backchannel" // "uh huh", "yeah" - continue AI
  | "soft_barge" // "wait", "hold on" - pause AI
  | "hard_barge" // Full interruption - stop AI
  | "unclear"; // Need more audio to classify

export type SpeechConfidence = "low" | "medium" | "high" | "very_high";

export type SupportedLanguage = "en" | "ar" | "es" | "fr" | "de" | "zh" | "ja" | "ko" | "pt" | "ru" | "hi" | "tr";

export interface BargeInEvent {
  id: string;
  type: BargeInClassification;
  timestamp: number;
  interruptedContent: string;
  interruptedAtWord: number;
  totalWords: number;
  completionPercentage: number;
  userTranscript?: string;
  resumable: boolean;
  contextSummary?: string; // Summary of truncated content for resumption
  activeToolCall?: ToolCallState; // Tool call that was interrupted
  language: SupportedLanguage;
}

export interface ToolCallState {
  id: string;
  name: string;
  status: "pending" | "executing" | "completed" | "cancelled" | "rolled_back";
  safeToInterrupt: boolean;
  rollbackAction?: () => Promise<void>;
  startedAt: number;
}

export interface VADResult {
  isSpeech: boolean;
  confidence: number;
  onsetTimestamp: number;
  duration: number;
  energy: number;
  language?: SupportedLanguage;
  spectralFeatures?: {
    centroid: number;
    bandwidth: number;
    rolloff: number;
  };
}

export interface CalibrationResult {
  ambientNoiseLevel: number;
  recommendedVadThreshold: number;
  recommendedSilenceThreshold: number;
  environmentType: "quiet" | "moderate" | "noisy";
  calibratedAt: number;
}

export interface BargeInConfig {
  // Language settings
  language: SupportedLanguage;
  autoDetectLanguage: boolean;
  accentProfile?: string; // e.g., "en-US", "en-GB", "en-IN"

  // Detection thresholds (adaptive)
  speechOnsetConfidence: number; // Default: 0.7, adjusted per user
  speechConfirmMs: number; // Default: 100ms
  hardBargeMinDuration: number; // Default: 300ms

  // Audio behavior
  fadeOutDuration: number; // Default: 30ms
  softBargeFadeLevel: number; // Default: 0.2 (20%)
  softBargeWaitMs: number; // Default: 2000ms

  // Backchannel detection (language-aware)
  backchannelMaxDuration: number; // Default: 500ms
  backchannelPhrases: Map<SupportedLanguage, string[]>;

  // Echo cancellation
  echoSuppressionEnabled: boolean;
  echoCorrelationThreshold: number; // Default: 0.55

  // Adaptive settings
  adaptiveSilenceEnabled: boolean;
  minSilenceMs: number; // Default: 200ms
  maxSilenceMs: number; // Default: 800ms

  // Calibration
  calibrationEnabled: boolean;
  calibrationDurationMs: number; // Default: 3000ms

  // Personalization
  userId?: string;
  persistUserPreferences: boolean;

  // Offline fallback
  useOfflineVAD: boolean;
  offlineVADModel: "silero-lite" | "webrtc-vad";
  offlineTTSCacheEnabled: boolean;
  offlineTTSCacheSizeMB: number; // Default: 50MB

  // Privacy
  encryptAudioInTransit: boolean;
  anonymizeTelemetry: boolean;
  audioRetentionPolicy: "none" | "session" | "24h" | "7d";

  // Tool-call integration
  allowInterruptDuringToolCalls: boolean;
  toolCallInterruptBehavior: "queue" | "cancel" | "smart";
}

// User-specific persisted preferences
export interface UserBargeInPreferences {
  userId: string;
  vadSensitivity: number; // 0.0 - 1.0, adjusted from calibration
  silenceThreshold: number;
  preferredLanguage: SupportedLanguage;
  accentProfile?: string;
  backchannelFrequency: "low" | "normal" | "high";
  feedbackPreferences: FeedbackPreferences;
  calibrationHistory: CalibrationResult[];
  lastUpdated: number;
}

export interface FeedbackPreferences {
  visualFeedbackEnabled: boolean;
  visualFeedbackStyle: "pulse" | "border" | "icon" | "minimal";
  hapticFeedbackEnabled: boolean;
  hapticIntensity: "light" | "medium" | "strong";
  audioFeedbackEnabled: boolean;
  audioFeedbackType: "tone" | "voice" | "none";
  voicePromptAfterHardBarge: boolean;
  voicePromptText?: string; // e.g., "I'm listening"
}

Phase 1: Neural VAD Integration

Goal: Replace energy-based VAD with ML-based detection for <30ms speech onset detection

New Files to Create

Implementation: Silero VAD Wrapper with Language Support

// src/lib/sileroVAD/index.ts

/**
 * Silero VAD Integration with Multilingual Support
 *
 * Silero VAD is a neural network-based Voice Activity Detector that runs
 * in WebAssembly via ONNX Runtime Web. It provides:
 * - 95%+ accuracy on speech detection
 * - ~30ms latency for onset detection
 * - Robustness to background noise
 * - Language-agnostic core with language-specific tuning
 *
 * Model: silero_vad.onnx (~2MB) or silero_vad_lite.onnx (~500KB for offline)
 * Input: 512 samples at 16kHz (32ms chunks)
 * Output: Probability of speech (0-1)
 */

import * as ort from "onnxruntime-web";
import { SupportedLanguage } from "../types";
import { LANGUAGE_VAD_CONFIGS } from "./languageModels";

export interface SileroVADConfig {
  modelPath: string;
  sampleRate: number;
  windowSize: number;
  speechThreshold: number;
  silenceThreshold: number;
  minSpeechDuration: number;
  minSilenceDuration: number;
  language: SupportedLanguage;
  adaptiveThreshold: boolean;
  onSpeechStart?: (confidence: number, language?: SupportedLanguage) => void;
  onSpeechEnd?: (duration: number) => void;
  onVADResult?: (result: VADResult) => void;
  onCalibrationComplete?: (result: CalibrationResult) => void;
}

export interface VADResult {
  probability: number;
  isSpeech: boolean;
  timestamp: number;
  processingTime: number;
  detectedLanguage?: SupportedLanguage;
}

export interface CalibrationResult {
  ambientNoiseLevel: number;
  recommendedVadThreshold: number;
  recommendedSilenceThreshold: number;
  environmentType: "quiet" | "moderate" | "noisy";
  calibratedAt: number;
}

export class SileroVAD {
  private session: ort.InferenceSession | null = null;
  private config: SileroVADConfig;
  private state: Float32Array;
  private sr: BigInt64Array;
  private isLoaded = false;

  private speechStartTime: number | null = null;
  private consecutiveSpeechWindows = 0;
  private consecutiveSilenceWindows = 0;
  private isSpeaking = false;

  // Calibration state
  private isCalibrating = false;
  private calibrationSamples: number[] = [];
  private adaptedThreshold: number;

  constructor(config: Partial<SileroVADConfig> = {}) {
    const languageConfig = LANGUAGE_VAD_CONFIGS[config.language || "en"] || {};

    this.config = {
      modelPath: "/silero_vad.onnx",
      sampleRate: 16000,
      windowSize: 512,
      speechThreshold: 0.5,
      silenceThreshold: 0.35,
      minSpeechDuration: 64,
      minSilenceDuration: 100,
      language: "en",
      adaptiveThreshold: true,
      ...languageConfig,
      ...config,
    };

    this.adaptedThreshold = this.config.speechThreshold;
    this.state = new Float32Array(2 * 1 * 64);
    this.sr = new BigInt64Array([BigInt(this.config.sampleRate)]);
  }

  async initialize(): Promise<void> {
    if (this.isLoaded) return;

    try {
      ort.env.wasm.wasmPaths = "/";

      this.session = await ort.InferenceSession.create(this.config.modelPath, {
        executionProviders: ["wasm"],
        graphOptimizationLevel: "all",
      });

      this.isLoaded = true;
      console.log("[SileroVAD] Model loaded successfully");
    } catch (error) {
      console.error("[SileroVAD] Failed to load model:", error);
      throw error;
    }
  }

  /**
   * Start calibration phase to measure ambient noise
   * Call this at session start for ~3 seconds of silence
   */
  startCalibration(durationMs: number = 3000): void {
    this.isCalibrating = true;
    this.calibrationSamples = [];

    setTimeout(() => {
      this.finishCalibration();
    }, durationMs);
  }

  private finishCalibration(): void {
    this.isCalibrating = false;

    if (this.calibrationSamples.length === 0) {
      return;
    }

    const avgEnergy = this.calibrationSamples.reduce((a, b) => a + b, 0) / this.calibrationSamples.length;
    const maxEnergy = Math.max(...this.calibrationSamples);

    let environmentType: "quiet" | "moderate" | "noisy";
    let recommendedThreshold: number;

    if (avgEnergy < 0.01) {
      environmentType = "quiet";
      recommendedThreshold = 0.4;
    } else if (avgEnergy < 0.05) {
      environmentType = "moderate";
      recommendedThreshold = 0.55;
    } else {
      environmentType = "noisy";
      recommendedThreshold = 0.7;
    }

    this.adaptedThreshold = recommendedThreshold;

    const result: CalibrationResult = {
      ambientNoiseLevel: avgEnergy,
      recommendedVadThreshold: recommendedThreshold,
      recommendedSilenceThreshold: recommendedThreshold - 0.15,
      environmentType,
      calibratedAt: Date.now(),
    };

    this.config.onCalibrationComplete?.(result);
  }

  async process(audioData: Float32Array): Promise<VADResult> {
    if (!this.session) {
      throw new Error("VAD not initialized. Call initialize() first.");
    }

    const startTime = performance.now();

    // During calibration, collect energy samples
    if (this.isCalibrating) {
      const energy = this.computeEnergy(audioData);
      this.calibrationSamples.push(energy);
    }

    const inputTensor = new ort.Tensor("float32", audioData, [1, audioData.length]);
    const stateTensor = new ort.Tensor("float32", this.state, [2, 1, 64]);
    const srTensor = new ort.Tensor("int64", this.sr, [1]);

    const results = await this.session.run({
      input: inputTensor,
      state: stateTensor,
      sr: srTensor,
    });

    const probability = (results.output.data as Float32Array)[0];
    const newState = results.stateN.data as Float32Array;

    this.state.set(newState);

    const processingTime = performance.now() - startTime;
    const threshold = this.config.adaptiveThreshold ? this.adaptedThreshold : this.config.speechThreshold;
    const isSpeech = probability >= threshold;

    this.trackSpeechState(probability, isSpeech);

    const result: VADResult = {
      probability,
      isSpeech,
      timestamp: performance.now(),
      processingTime,
    };

    this.config.onVADResult?.(result);

    return result;
  }

  private computeEnergy(audioData: Float32Array): number {
    let sum = 0;
    for (let i = 0; i < audioData.length; i++) {
      sum += audioData[i] * audioData[i];
    }
    return Math.sqrt(sum / audioData.length);
  }

  private trackSpeechState(probability: number, isSpeech: boolean): void {
    const windowDuration = (this.config.windowSize / this.config.sampleRate) * 1000;

    if (isSpeech) {
      this.consecutiveSpeechWindows++;
      this.consecutiveSilenceWindows = 0;

      const speechDuration = this.consecutiveSpeechWindows * windowDuration;

      if (!this.isSpeaking && speechDuration >= this.config.minSpeechDuration) {
        this.isSpeaking = true;
        this.speechStartTime = performance.now() - speechDuration;
        this.config.onSpeechStart?.(probability, this.config.language);
      }
    } else {
      this.consecutiveSilenceWindows++;

      const silenceDuration = this.consecutiveSilenceWindows * windowDuration;

      if (this.isSpeaking && silenceDuration >= this.config.minSilenceDuration) {
        const totalDuration = performance.now() - (this.speechStartTime || 0);
        this.isSpeaking = false;
        this.speechStartTime = null;
        this.consecutiveSpeechWindows = 0;
        this.config.onSpeechEnd?.(totalDuration);
      }
    }
  }

  setLanguage(language: SupportedLanguage): void {
    this.config.language = language;
    const languageConfig = LANGUAGE_VAD_CONFIGS[language];
    if (languageConfig) {
      this.config.speechThreshold = languageConfig.speechThreshold ?? this.config.speechThreshold;
      this.config.minSpeechDuration = languageConfig.minSpeechDuration ?? this.config.minSpeechDuration;
    }
  }

  updateThreshold(threshold: number): void {
    this.adaptedThreshold = Math.max(0.3, Math.min(0.9, threshold));
  }

  reset(): void {
    this.state.fill(0);
    this.isSpeaking = false;
    this.speechStartTime = null;
    this.consecutiveSpeechWindows = 0;
    this.consecutiveSilenceWindows = 0;
  }

  destroy(): void {
    this.session?.release();
    this.session = null;
    this.isLoaded = false;
  }
}

Language-Specific VAD Configurations

// src/lib/sileroVAD/languageModels.ts

import { SupportedLanguage } from "../types";

interface LanguageVADConfig {
  speechThreshold?: number;
  silenceThreshold?: number;
  minSpeechDuration?: number;
  minSilenceDuration?: number;
  // Some languages have longer pauses between words
  pauseTolerance?: number;
}

export const LANGUAGE_VAD_CONFIGS: Record<SupportedLanguage, LanguageVADConfig> = {
  en: {
    speechThreshold: 0.5,
    minSpeechDuration: 64,
    minSilenceDuration: 100,
  },
  ar: {
    // Arabic has emphatic consonants that may need higher threshold
    speechThreshold: 0.55,
    minSpeechDuration: 80,
    minSilenceDuration: 120,
    pauseTolerance: 150,
  },
  es: {
    speechThreshold: 0.48,
    minSpeechDuration: 60,
    minSilenceDuration: 90,
  },
  fr: {
    speechThreshold: 0.5,
    minSpeechDuration: 64,
    minSilenceDuration: 100,
  },
  de: {
    // German has longer compound words
    speechThreshold: 0.52,
    minSpeechDuration: 70,
    minSilenceDuration: 110,
  },
  zh: {
    // Mandarin tones require careful threshold
    speechThreshold: 0.55,
    minSpeechDuration: 80,
    minSilenceDuration: 120,
  },
  ja: {
    speechThreshold: 0.5,
    minSpeechDuration: 64,
    minSilenceDuration: 100,
  },
  ko: {
    speechThreshold: 0.52,
    minSpeechDuration: 70,
    minSilenceDuration: 110,
  },
  pt: {
    speechThreshold: 0.48,
    minSpeechDuration: 60,
    minSilenceDuration: 90,
  },
  ru: {
    speechThreshold: 0.52,
    minSpeechDuration: 70,
    minSilenceDuration: 110,
  },
  hi: {
    speechThreshold: 0.55,
    minSpeechDuration: 80,
    minSilenceDuration: 120,
  },
  tr: {
    speechThreshold: 0.5,
    minSpeechDuration: 64,
    minSilenceDuration: 100,
  },
};

Implementation: useNeuralVAD Hook

// src/hooks/useNeuralVAD.ts

import { useCallback, useEffect, useRef, useState } from "react";
import { SileroVAD, VADResult, SileroVADConfig, CalibrationResult } from "../lib/sileroVAD";
import { SupportedLanguage, UserBargeInPreferences } from "../lib/types";

export interface UseNeuralVADOptions {
  enabled?: boolean;
  language?: SupportedLanguage;
  autoCalibrate?: boolean;
  userPreferences?: UserBargeInPreferences;
  onSpeechStart?: (confidence: number, language?: SupportedLanguage) => void;
  onSpeechEnd?: (duration: number) => void;
  onVADResult?: (result: VADResult) => void;
  onCalibrationComplete?: (result: CalibrationResult) => void;
  config?: Partial<SileroVADConfig>;
}

export interface UseNeuralVADReturn {
  isLoaded: boolean;
  isListening: boolean;
  isSpeaking: boolean;
  isCalibrating: boolean;
  currentConfidence: number;
  calibrationResult: CalibrationResult | null;
  startListening: (stream: MediaStream) => Promise<void>;
  stopListening: () => void;
  startCalibration: (durationMs?: number) => void;
  setLanguage: (language: SupportedLanguage) => void;
  updateThreshold: (threshold: number) => void;
  processAudioChunk: (data: Float32Array) => Promise<VADResult | null>;
}

export function useNeuralVAD(options: UseNeuralVADOptions = {}): UseNeuralVADReturn {
  const {
    enabled = true,
    language = "en",
    autoCalibrate = true,
    userPreferences,
    onSpeechStart,
    onSpeechEnd,
    onVADResult,
    onCalibrationComplete,
    config = {},
  } = options;

  const [isLoaded, setIsLoaded] = useState(false);
  const [isListening, setIsListening] = useState(false);
  const [isSpeaking, setIsSpeaking] = useState(false);
  const [isCalibrating, setIsCalibrating] = useState(false);
  const [currentConfidence, setCurrentConfidence] = useState(0);
  const [calibrationResult, setCalibrationResult] = useState<CalibrationResult | null>(null);

  const vadRef = useRef<SileroVAD | null>(null);
  const audioContextRef = useRef<AudioContext | null>(null);
  const workletNodeRef = useRef<AudioWorkletNode | null>(null);
  const streamRef = useRef<MediaStream | null>(null);

  // Apply user preferences if available
  const effectiveConfig = {
    ...config,
    language,
    speechThreshold: userPreferences?.vadSensitivity ?? config.speechThreshold,
  };

  useEffect(() => {
    if (!enabled) return;

    const vad = new SileroVAD({
      ...effectiveConfig,
      onSpeechStart: (confidence, detectedLang) => {
        setIsSpeaking(true);
        onSpeechStart?.(confidence, detectedLang);
      },
      onSpeechEnd: (duration) => {
        setIsSpeaking(false);
        onSpeechEnd?.(duration);
      },
      onVADResult: (result) => {
        setCurrentConfidence(result.probability);
        onVADResult?.(result);
      },
      onCalibrationComplete: (result) => {
        setIsCalibrating(false);
        setCalibrationResult(result);
        onCalibrationComplete?.(result);
      },
    });

    vadRef.current = vad;

    vad
      .initialize()
      .then(() => setIsLoaded(true))
      .catch((error) => console.error("[useNeuralVAD] Failed to initialize:", error));

    return () => {
      vad.destroy();
      vadRef.current = null;
    };
  }, [enabled, language]);

  const startCalibration = useCallback((durationMs: number = 3000) => {
    if (!vadRef.current) return;
    setIsCalibrating(true);
    vadRef.current.startCalibration(durationMs);
  }, []);

  const startListening = useCallback(
    async (stream: MediaStream) => {
      if (!vadRef.current || !isLoaded) {
        throw new Error("VAD not ready");
      }

      const audioContext = new AudioContext({ sampleRate: 16000 });
      audioContextRef.current = audioContext;
      streamRef.current = stream;

      await audioContext.audioWorklet.addModule("/vad-processor.js");

      const source = audioContext.createMediaStreamSource(stream);
      const workletNode = new AudioWorkletNode(audioContext, "vad-processor", {
        processorOptions: { windowSize: 512 },
      });

      workletNode.port.onmessage = async (event) => {
        if (event.data.type === "audio") {
          const audioData = new Float32Array(event.data.samples);
          await vadRef.current?.process(audioData);
        }
      };

      source.connect(workletNode);
      workletNodeRef.current = workletNode;
      setIsListening(true);

      // Auto-calibrate on first listen if enabled
      if (autoCalibrate && !calibrationResult) {
        startCalibration();
      }
    },
    [isLoaded, autoCalibrate, calibrationResult, startCalibration],
  );

  const stopListening = useCallback(() => {
    workletNodeRef.current?.disconnect();
    audioContextRef.current?.close();
    streamRef.current?.getTracks().forEach((track) => track.stop());

    vadRef.current?.reset();
    setIsListening(false);
    setIsSpeaking(false);
  }, []);

  const setLanguage = useCallback((lang: SupportedLanguage) => {
    vadRef.current?.setLanguage(lang);
  }, []);

  const updateThreshold = useCallback((threshold: number) => {
    vadRef.current?.updateThreshold(threshold);
  }, []);

  const processAudioChunk = useCallback(
    async (data: Float32Array) => {
      if (!vadRef.current || !isLoaded) return null;
      return vadRef.current.process(data);
    },
    [isLoaded],
  );

  return {
    isLoaded,
    isListening,
    isSpeaking,
    isCalibrating,
    currentConfidence,
    calibrationResult,
    startListening,
    stopListening,
    startCalibration,
    setLanguage,
    updateThreshold,
    processAudioChunk,
  };
}

Files to Modify

File: apps/web-app/package.json

{
  "dependencies": {
    "onnxruntime-web": "^1.17.0"
  }
}

File: apps/web-app/src/hooks/useThinkerTalkerSession.ts

• Import and integrate useNeuralVAD
• Add handleBargeInDetected function
• Modify audio processing to use neural VAD
• Integrate offline fallback logic

Phase 2: Instant Response & Feedback

Goal: User knows their interruption was heard within 50ms with configurable feedback

New Files to Create

Implementation: Enhanced BargeInFeedback Component

// src/components/voice/BargeInFeedback.tsx

import { useEffect, useState, useMemo } from 'react';
import { motion, AnimatePresence } from 'framer-motion';
import { FeedbackPreferences } from '../../lib/types';
import { useHapticFeedback } from '../../hooks/useHapticFeedback';
import { playAudioFeedback, speakPrompt } from '../../lib/audioFeedback';

interface BargeInFeedbackProps {
  isActive: boolean;
  type: 'detected' | 'confirmed' | 'backchannel' | 'soft' | 'hard';
  confidence?: number;
  preferences: FeedbackPreferences;
  onAnimationComplete?: () => void;
}

export function BargeInFeedback({
  isActive,
  type,
  confidence = 0,
  preferences,
  onAnimationComplete,
}: BargeInFeedbackProps) {
  const [showPulse, setShowPulse] = useState(false);
  const { triggerHaptic } = useHapticFeedback();

  const pulseColors = useMemo(() => ({
    detected: 'rgba(59, 130, 246, 0.5)',
    confirmed: 'rgba(34, 197, 94, 0.5)',
    backchannel: 'rgba(168, 162, 158, 0.3)',
    soft: 'rgba(251, 191, 36, 0.5)',
    hard: 'rgba(239, 68, 68, 0.5)',
  }), []);

  const hapticMap = useMemo(() => ({
    detected: 'bargeInDetected',
    confirmed: 'bargeInConfirmed',
    backchannel: 'backchannel',
    soft: 'softBarge',
    hard: 'hardBarge',
  } as const), []);

  useEffect(() => {
    if (isActive) {
      // Visual feedback
      if (preferences.visualFeedbackEnabled) {
        setShowPulse(true);
        const timer = setTimeout(() => {
          setShowPulse(false);
          onAnimationComplete?.();
        }, 300);
        return () => clearTimeout(timer);
      }

      // Haptic feedback
      if (preferences.hapticFeedbackEnabled) {
        triggerHaptic(hapticMap[type], preferences.hapticIntensity);
      }

      // Audio feedback
      if (preferences.audioFeedbackEnabled) {
        if (preferences.audioFeedbackType === 'tone') {
          playAudioFeedback(type);
        } else if (preferences.audioFeedbackType === 'voice' && type === 'hard') {
          if (preferences.voicePromptAfterHardBarge) {
            speakPrompt(preferences.voicePromptText || "I'm listening");
          }
        }
      }
    }
  }, [isActive, type, preferences, triggerHaptic, hapticMap, onAnimationComplete]);

  if (!preferences.visualFeedbackEnabled) {
    return null;
  }

  const renderFeedback = () => {
    switch (preferences.visualFeedbackStyle) {
      case 'pulse':
        return (
          <motion.div
            className="fixed inset-0 pointer-events-none z-50"
            initial={{ opacity: 0 }}
            animate={{ opacity: 1 }}
            exit={{ opacity: 0 }}
            transition={{ duration: 0.05 }}
          >
            <motion.div
              className="absolute top-1/2 left-1/2 -translate-x-1/2 -translate-y-1/2 rounded-full"
              style={{
                backgroundColor: pulseColors[type],
                boxShadow: `0 0 60px 30px ${pulseColors[type]}`,
              }}
              initial={{ width: 20, height: 20, opacity: 0.8 }}
              animate={{ width: 200, height: 200, opacity: 0 }}
              transition={{ duration: 0.3, ease: 'easeOut' }}
            />
          </motion.div>
        );

      case 'border':
        return (
          <motion.div
            className="fixed inset-0 pointer-events-none z-50 border-4 rounded-lg"
            style={{ borderColor: pulseColors[type] }}
            initial={{ opacity: 0 }}
            animate={{ opacity: 1 }}
            exit={{ opacity: 0 }}
            transition={{ duration: 0.1 }}
          />
        );

      case 'icon':
        return (
          <motion.div
            className="fixed top-4 right-4 pointer-events-none z-50"
            initial={{ scale: 0, opacity: 0 }}
            animate={{ scale: 1, opacity: 1 }}
            exit={{ scale: 0, opacity: 0 }}
          >
            <div
              className="w-12 h-12 rounded-full flex items-center justify-center"
              style={{ backgroundColor: pulseColors[type] }}
            >
              {type === 'hard' && '✋'}
              {type === 'soft' && '⏸'}
              {type === 'backchannel' && '👂'}
              {type === 'detected' && '🎤'}
              {type === 'confirmed' && '✓'}
            </div>
          </motion.div>
        );

      case 'minimal':
        return (
          <motion.div
            className="fixed bottom-20 left-1/2 -translate-x-1/2 pointer-events-none z-50"
            initial={{ opacity: 0, y: 10 }}
            animate={{ opacity: 1, y: 0 }}
            exit={{ opacity: 0, y: -10 }}
          >
            <div
              className="w-2 h-2 rounded-full"
              style={{ backgroundColor: pulseColors[type] }}
            />
          </motion.div>
        );
    }
  };

  return (
    <AnimatePresence>
      {showPulse && renderFeedback()}
    </AnimatePresence>
  );
}

Implementation: Enhanced Haptic Feedback Hook

// src/hooks/useHapticFeedback.ts

import { useCallback, useEffect, useRef } from "react";

type HapticIntensity = "light" | "medium" | "strong";
type HapticType =
  | "bargeInDetected"
  | "bargeInConfirmed"
  | "backchannel"
  | "softBarge"
  | "hardBarge"
  | "speechStart"
  | "error"
  | "calibrationComplete";

const HAPTIC_PATTERNS: Record<HapticType, Record<HapticIntensity, number[]>> = {
  bargeInDetected: {
    light: [10, 20, 10],
    medium: [15, 30, 15],
    strong: [25, 40, 25],
  },
  bargeInConfirmed: {
    light: [25],
    medium: [40],
    strong: [60],
  },
  backchannel: {
    light: [3],
    medium: [5],
    strong: [10],
  },
  softBarge: {
    light: [15, 30, 15],
    medium: [25, 50, 25],
    strong: [40, 70, 40],
  },
  hardBarge: {
    light: [30, 20, 30],
    medium: [50, 30, 50],
    strong: [80, 40, 80],
  },
  speechStart: {
    light: [5],
    medium: [10],
    strong: [15],
  },
  error: {
    light: [50, 30, 50, 30, 50],
    medium: [100, 50, 100, 50, 100],
    strong: [150, 70, 150, 70, 150],
  },
  calibrationComplete: {
    light: [20, 100, 20],
    medium: [30, 100, 30],
    strong: [50, 100, 50],
  },
};

export function useHapticFeedback() {
  const isSupported = useRef(false);

  useEffect(() => {
    isSupported.current = "vibrate" in navigator;
  }, []);

  const vibrate = useCallback((pattern: number | number[]) => {
    if (!isSupported.current) return false;
    try {
      navigator.vibrate(pattern);
      return true;
    } catch {
      return false;
    }
  }, []);

  const triggerHaptic = useCallback(
    (type: HapticType, intensity: HapticIntensity = "medium") => {
      const pattern = HAPTIC_PATTERNS[type]?.[intensity];
      if (pattern) vibrate(pattern);
    },
    [vibrate],
  );

  const stopHaptic = useCallback(() => {
    if (isSupported.current) {
      navigator.vibrate(0);
    }
  }, []);

  return {
    isSupported: isSupported.current,
    triggerHaptic,
    stopHaptic,
  };
}

Implementation: Audio Feedback with Voice Prompts

// src/lib/audioFeedback.ts

type FeedbackType = "detected" | "confirmed" | "backchannel" | "soft" | "hard";

const audioContext = new (window.AudioContext || (window as any).webkitAudioContext)();

const TONE_FREQUENCIES: Record<FeedbackType, number> = {
  detected: 440, // A4
  confirmed: 523.25, // C5
  backchannel: 329.63, // E4
  soft: 392, // G4
  hard: 587.33, // D5
};

const TONE_DURATIONS: Record<FeedbackType, number> = {
  detected: 50,
  confirmed: 80,
  backchannel: 30,
  soft: 60,
  hard: 100,
};

export function playAudioFeedback(type: FeedbackType, volume: number = 0.3): void {
  const oscillator = audioContext.createOscillator();
  const gainNode = audioContext.createGain();

  oscillator.connect(gainNode);
  gainNode.connect(audioContext.destination);

  oscillator.frequency.value = TONE_FREQUENCIES[type];
  oscillator.type = "sine";

  gainNode.gain.setValueAtTime(volume, audioContext.currentTime);
  gainNode.gain.exponentialRampToValueAtTime(0.001, audioContext.currentTime + TONE_DURATIONS[type] / 1000);

  oscillator.start(audioContext.currentTime);
  oscillator.stop(audioContext.currentTime + TONE_DURATIONS[type] / 1000);
}

let speechSynthesis: SpeechSynthesis | null = null;

export function speakPrompt(text: string, language: string = "en-US"): void {
  if (!speechSynthesis) {
    speechSynthesis = window.speechSynthesis;
  }

  // Cancel any ongoing speech
  speechSynthesis.cancel();

  const utterance = new SpeechSynthesisUtterance(text);
  utterance.lang = language;
  utterance.rate = 1.1;
  utterance.pitch = 1.0;
  utterance.volume = 0.8;

  speechSynthesis.speak(utterance);
}

// Pre-load common voice prompts for faster playback
const VOICE_PROMPT_CACHE = new Map<string, AudioBuffer>();

export async function preloadVoicePrompt(text: string, language: string = "en-US"): Promise<void> {
  // Use Web Speech API to pre-synthesize
  // In production, use pre-recorded audio files or TTS API
  const cacheKey = `${language}:${text}`;
  if (VOICE_PROMPT_CACHE.has(cacheKey)) return;

  // Placeholder for pre-recorded audio loading
  // const response = await fetch(`/audio/prompts/${language}/${encodeURIComponent(text)}.mp3`);
  // const arrayBuffer = await response.arrayBuffer();
  // const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
  // VOICE_PROMPT_CACHE.set(cacheKey, audioBuffer);
}

export function playVoicePromptFromCache(text: string, language: string = "en-US"): boolean {
  const cacheKey = `${language}:${text}`;
  const buffer = VOICE_PROMPT_CACHE.get(cacheKey);

  if (!buffer) return false;

  const source = audioContext.createBufferSource();
  source.buffer = buffer;
  source.connect(audioContext.destination);
  source.start(0);

  return true;
}

Phase 3: Context-Aware Interruption Intelligence

Goal: Understand the intent behind interruptions with multilingual support

New Files to Create

Multilingual Backchannel Patterns

// src/lib/bargeInClassifier/phraseLibrary.ts

import { SupportedLanguage } from "../types";

export interface BackchannelPattern {
  phrases: string[];
  maxDuration: number;
  confidence?: number;
}

export interface SoftBargePattern {
  phrases: string[];
  requiresFollowUp: boolean;
}

export const BACKCHANNEL_PATTERNS: Record<SupportedLanguage, BackchannelPattern[]> = {
  en: [
    { phrases: ["uh huh", "uh-huh", "uhuh", "mm hmm", "mmhmm", "mhm"], maxDuration: 600 },
    { phrases: ["yeah", "yep", "yes", "yea", "ya"], maxDuration: 400 },
    { phrases: ["okay", "ok", "k", "kay"], maxDuration: 400 },
    { phrases: ["right", "right right"], maxDuration: 500 },
    { phrases: ["sure", "got it", "gotcha"], maxDuration: 500 },
    { phrases: ["I see", "interesting", "cool"], maxDuration: 600 },
  ],
  ar: [
    { phrases: ["نعم", "اه", "اها", "ايوه", "ايه"], maxDuration: 500 },
    { phrases: ["صح", "صحيح", "تمام", "ماشي"], maxDuration: 500 },
    { phrases: ["طيب", "حسنا", "اوكي"], maxDuration: 400 },
    { phrases: ["فاهم", "مفهوم"], maxDuration: 600 },
  ],
  es: [
    { phrases: ["sí", "si", "ajá", "aha"], maxDuration: 400 },
    { phrases: ["vale", "ok", "bueno"], maxDuration: 400 },
    { phrases: ["claro", "entiendo", "ya"], maxDuration: 500 },
    { phrases: ["mmm", "mhm"], maxDuration: 400 },
  ],
  fr: [
    { phrases: ["oui", "ouais", "mouais"], maxDuration: 400 },
    { phrases: ["d'accord", "ok", "entendu"], maxDuration: 500 },
    { phrases: ["je vois", "ah bon", "mmm"], maxDuration: 600 },
    { phrases: ["bien", "super", "parfait"], maxDuration: 500 },
  ],
  de: [
    { phrases: ["ja", "jap", "jo"], maxDuration: 400 },
    { phrases: ["okay", "ok", "gut"], maxDuration: 400 },
    { phrases: ["genau", "richtig", "stimmt"], maxDuration: 500 },
    { phrases: ["verstehe", "aha", "mmm"], maxDuration: 600 },
  ],
  zh: [
    { phrases: ["嗯", "哦", "啊"], maxDuration: 400 },
    { phrases: ["是", "对", "好"], maxDuration: 400 },
    { phrases: ["明白", "了解", "知道"], maxDuration: 600 },
    { phrases: ["没问题", "可以"], maxDuration: 600 },
  ],
  ja: [
    { phrases: ["はい", "うん", "ええ"], maxDuration: 400 },
    { phrases: ["そうですね", "なるほど"], maxDuration: 700 },
    { phrases: ["分かりました", "了解"], maxDuration: 800 },
  ],
  ko: [
    { phrases: ["네", "응", "예"], maxDuration: 400 },
    { phrases: ["그래요", "맞아요", "알겠어요"], maxDuration: 600 },
    { phrases: ["좋아요", "오케이"], maxDuration: 500 },
  ],
  pt: [
    { phrases: ["sim", "é", "ahã"], maxDuration: 400 },
    { phrases: ["ok", "tá", "certo"], maxDuration: 400 },
    { phrases: ["entendi", "compreendo", "sei"], maxDuration: 600 },
  ],
  ru: [
    { phrases: ["да", "ага", "угу"], maxDuration: 400 },
    { phrases: ["понятно", "ясно", "хорошо"], maxDuration: 600 },
    { phrases: ["ладно", "окей", "ок"], maxDuration: 400 },
  ],
  hi: [
    { phrases: ["हाँ", "जी", "अच्छा"], maxDuration: 400 },
    { phrases: ["ठीक है", "समझ गया", "सही"], maxDuration: 600 },
    { phrases: ["हम्म", "ओके"], maxDuration: 400 },
  ],
  tr: [
    { phrases: ["evet", "hı hı", "tamam"], maxDuration: 400 },
    { phrases: ["anladım", "peki", "oldu"], maxDuration: 600 },
    { phrases: ["doğru", "iyi", "güzel"], maxDuration: 500 },
  ],
};

export const SOFT_BARGE_PATTERNS: Record<SupportedLanguage, SoftBargePattern[]> = {
  en: [
    { phrases: ["wait", "hold on", "hang on", "one moment"], requiresFollowUp: true },
    { phrases: ["actually", "but", "well", "um"], requiresFollowUp: true },
    { phrases: ["let me", "can I", "I want to"], requiresFollowUp: true },
  ],
  ar: [
    { phrases: ["انتظر", "لحظة", "ثانية"], requiresFollowUp: true },
    { phrases: ["بس", "لكن", "في الحقيقة"], requiresFollowUp: true },
  ],
  es: [
    { phrases: ["espera", "un momento", "para"], requiresFollowUp: true },
    { phrases: ["pero", "en realidad", "bueno"], requiresFollowUp: true },
  ],
  fr: [
    { phrases: ["attends", "un moment", "une seconde"], requiresFollowUp: true },
    { phrases: ["mais", "en fait", "euh"], requiresFollowUp: true },
  ],
  de: [
    { phrases: ["warte", "moment", "einen Augenblick"], requiresFollowUp: true },
    { phrases: ["aber", "eigentlich", "also"], requiresFollowUp: true },
  ],
  zh: [
    { phrases: ["等一下", "等等", "稍等"], requiresFollowUp: true },
    { phrases: ["但是", "其实", "不过"], requiresFollowUp: true },
  ],
  ja: [
    { phrases: ["ちょっと待って", "待って", "少々"], requiresFollowUp: true },
    { phrases: ["でも", "実は", "あの"], requiresFollowUp: true },
  ],
  ko: [
    { phrases: ["잠깐만", "잠시만요", "기다려"], requiresFollowUp: true },
    { phrases: ["그런데", "사실은", "근데"], requiresFollowUp: true },
  ],
  pt: [
    { phrases: ["espera", "um momento", "peraí"], requiresFollowUp: true },
    { phrases: ["mas", "na verdade", "bom"], requiresFollowUp: true },
  ],
  ru: [
    { phrases: ["подожди", "секунду", "минутку"], requiresFollowUp: true },
    { phrases: ["но", "на самом деле", "вообще-то"], requiresFollowUp: true },
  ],
  hi: [
    { phrases: ["रुको", "एक मिनट", "ज़रा"], requiresFollowUp: true },
    { phrases: ["लेकिन", "असल में", "वैसे"], requiresFollowUp: true },
  ],
  tr: [
    { phrases: ["bekle", "bir dakika", "dur"], requiresFollowUp: true },
    { phrases: ["ama", "aslında", "şey"], requiresFollowUp: true },
  ],
};