2:I[7012,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],"MarkdownRenderer"] 4:I[9856,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],""] 5:I[4126,[],""] 7:I[9630,[],""] 8:I[4278,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"HeadingProvider"] 9:I[1476,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Header"] a:I[3167,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Sidebar"] b:I[7409,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"PageFrame"] 3:T1904, # Backend Debugging Guide **Last Updated:** 2025-11-27 **Component:** `services/api-gateway/` --- ## Symptoms ### 500 Internal Server Error **Likely Causes:** - Unhandled exception in request handler - Database connection timeout - Missing required environment variable - External service failure (OpenAI, Qdrant) **Steps to Investigate:** 1. Check API Gateway logs (Docker container): ```bash docker logs voiceassist-server --tail 100 2>&1 | grep -i error ``` 2. Look for stack traces: ```bash docker logs voiceassist-server --since "10m" 2>&1 | grep -A 20 "Traceback" ``` 3. Check health endpoints: ```bash curl http://localhost:8000/health curl http://localhost:8000/ready ``` 4. Verify environment variables: ```bash # Check if critical vars are set in .env grep -E "DATABASE_URL|REDIS_URL|OPENAI_API_KEY" /home/asimo/VoiceAssist/.env ``` **Relevant Logs:** - `docker logs voiceassist-server` - Structured JSON logs with `trace_id` **Relevant Code Paths:** - `services/api-gateway/app/main.py` - Exception handlers - `services/api-gateway/app/core/exceptions.py` - Custom exceptions - `services/api-gateway/app/api/*.py` - Route handlers --- ### 401 Unauthorized **Likely Causes:** - JWT token expired - Token missing from request - Token signed with wrong key - User revoked or deactivated **Steps to Investigate:** 1. Decode the JWT (without verifying): ```bash # Extract token from Authorization header echo "YOUR_JWT_TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null | jq . ``` 2. Check token expiration: ```bash # Look at 'exp' claim - Unix timestamp ``` 3. Verify JWT secret matches: ```bash # Compare JWT_SECRET_KEY in env with what was used to sign ``` 4. Check if user is active: ```sql SELECT id, email, is_active FROM users WHERE id = 'USER_ID'; ``` **Relevant Code Paths:** - `services/api-gateway/app/core/security.py` - JWT verification - `services/api-gateway/app/core/dependencies.py` - Auth dependencies --- ### 503 Service Unavailable **Likely Causes:** - Database connection pool exhausted - Redis not responding - Qdrant vector store down - External API rate limited **Steps to Investigate:** 1. Check database connectivity: ```bash # PostgreSQL psql -h localhost -U voiceassist -d voiceassist -c "SELECT 1" # Check connection count psql -c "SELECT count(*) FROM pg_stat_activity WHERE datname = 'voiceassist'" ``` 2. Check Redis: ```bash redis-cli ping redis-cli info clients ``` 3. Check Qdrant: ```bash curl http://localhost:6333/collections ``` 4. Check API Gateway connection pool: ```bash curl http://localhost:8000/metrics | grep "db_connection" ``` **Relevant Code Paths:** - `services/api-gateway/app/core/database.py` - DB connection - `services/api-gateway/app/services/cache_service.py` - Redis - `services/api-gateway/app/services/vector_store_service.py` - Qdrant --- ## Database Issues ### Connection Pool Exhaustion **Symptoms:** - Requests hanging - Timeout errors - "too many connections" in logs **Investigation:** ```bash # Check active connections psql -c "SELECT count(*), state FROM pg_stat_activity WHERE datname = 'voiceassist' GROUP BY state" # Find long-running queries psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE datname = 'voiceassist' AND state != 'idle' ORDER BY duration DESC" # Kill stuck query if needed psql -c "SELECT pg_terminate_backend(PID)" ``` **Fix:** - Increase `pool_size` in database config - Add connection timeout - Check for leaked connections in code ### Migration Issues ```bash # Check current migration version cd services/api-gateway alembic current # Check migration history alembic history # Run pending migrations alembic upgrade head # Rollback if needed alembic downgrade -1 ``` --- ## Cache Issues ### Redis Not Responding **Symptoms:** - Cache misses everywhere - Slower response times - Session lookup failures **Investigation:** ```bash # Check Redis status sudo systemctl status redis-server redis-cli ping # Check memory redis-cli info memory # Check connected clients redis-cli client list # Check slow log redis-cli slowlog get 10 ``` **Relevant Code Paths:** - `services/api-gateway/app/services/cache_service.py` - `services/api-gateway/app/core/config.py` - REDIS_URL --- ## OpenAI API Issues ### Rate Limiting / Quota Exceeded **Symptoms:** - 429 errors from OpenAI - Empty AI responses - Timeout waiting for completion **Investigation:** 1. Check recent OpenAI calls: ```bash docker logs voiceassist-server --since "1h" 2>&1 | grep -i "openai\|rate\|429" ``` 2. Check API key validity: ```bash curl https://api.openai.com/v1/models \ -H "Authorization: Bearer $OPENAI_API_KEY" ``` 3. Check usage dashboard: https://platform.openai.com/usage **Relevant Code Paths:** - `services/api-gateway/app/services/llm_client.py` - `services/api-gateway/app/core/config.py` - OPENAI_API_KEY --- ## RAG Pipeline Issues ### Poor Search Results **Symptoms:** - Irrelevant document retrieval - Empty results for valid queries - Low confidence scores **Investigation:** 1. Check vector store health: ```bash curl http://localhost:6333/collections/medical_docs ``` 2. Test embedding generation: ```python # In Python shell from app.services.embedding_service import EmbeddingService svc = EmbeddingService() embedding = await svc.embed_text("test query") print(len(embedding)) # Should be 1536 for OpenAI ada-002 ``` 3. Check document count: ```bash curl http://localhost:6333/collections/medical_docs | jq '.result.points_count' ``` **Relevant Code Paths:** - `services/api-gateway/app/services/rag_service.py` - `services/api-gateway/app/services/embedding_service.py` - `services/api-gateway/app/services/vector_store_service.py` --- ## Metrics to Monitor | Metric | Normal Range | Alert Threshold | | --------------------------------- | ------------ | --------------- | | `http_request_duration_seconds` | < 500ms | > 2s | | `db_connection_pool_size` | 5-20 | > 80% used | | `http_requests_total{status=5xx}` | 0 | > 10/min | | `redis_connection_errors` | 0 | > 0 | --- ## Related Documentation - [Debugging Overview](./DEBUGGING_OVERVIEW.md) - [Backend Architecture](../BACKEND_ARCHITECTURE.md) - [API Reference](../API_REFERENCE.md) 6:["slug","debugging/DEBUGGING_BACKEND","c"] 0:["X7oMT3VrOffzp0qvbeOas",[[["",{"children":["docs",{"children":[["slug","debugging/DEBUGGING_BACKEND","c"],{"children":["__PAGE__?{\"slug\":[\"debugging\",\"DEBUGGING_BACKEND\"]}",{}]}]}]},"$undefined","$undefined",true],["",{"children":["docs",{"children":[["slug","debugging/DEBUGGING_BACKEND","c"],{"children":["__PAGE__",{},[["$L1",["$","div",null,{"children":[["$","div",null,{"className":"mb-6 flex items-center justify-between gap-4","children":[["$","div",null,{"children":[["$","p",null,{"className":"text-sm text-gray-500 dark:text-gray-400","children":"Docs / Raw"}],["$","h1",null,{"className":"text-3xl font-bold text-gray-900 dark:text-white","children":"Backend Debugging Guide"}],["$","p",null,{"className":"text-sm text-gray-600 dark:text-gray-400","children":["Sourced from"," ",["$","code",null,{"className":"font-mono text-xs","children":["docs/","debugging/DEBUGGING_BACKEND.md"]}]]}]]}],["$","a",null,{"href":"https://github.com/mohammednazmy/VoiceAssist/edit/main/docs/debugging/DEBUGGING_BACKEND.md","target":"_blank","rel":"noreferrer","className":"inline-flex items-center gap-2 rounded-md border border-gray-200 dark:border-gray-700 px-3 py-1.5 text-sm text-gray-700 dark:text-gray-200 hover:border-primary-500 dark:hover:border-primary-400 hover:text-primary-700 dark:hover:text-primary-300","children":"Edit on GitHub"}]]}],["$","div",null,{"className":"rounded-lg border border-gray-200 dark:border-gray-800 bg-white dark:bg-gray-900 p-6","children":["$","$L2",null,{"content":"$3"}]}],["$","div",null,{"className":"mt-6 flex flex-wrap gap-2 text-sm","children":[["$","$L4",null,{"href":"/reference/all-docs","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"← All documentation"}],["$","$L4",null,{"href":"/","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"Home"}]]}]]}],null],null],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children","$6","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/_next/static/css/7f586cdbbaa33ff7.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","className":"h-full","children":["$","body",null,{"className":"__className_f367f3 h-full bg-white dark:bg-gray-900","children":[["$","a",null,{"href":"#main-content","className":"skip-to-content","children":"Skip to main content"}],["$","$L8",null,{"children":[["$","$L9",null,{}],["$","$La",null,{}],["$","main",null,{"id":"main-content","className":"lg:pl-64","role":"main","aria-label":"Documentation content","children":["$","$Lb",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]]}]]}]}]],null],null],["$Lc",null]]]] c:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"Backend Debugging Guide | Docs | VoiceAssist Docs"}],["$","meta","3",{"name":"description","content":"Debug API Gateway, database, cache, and backend services in VoiceAssist."}],["$","meta","4",{"name":"keywords","content":"VoiceAssist,documentation,medical AI,voice assistant,healthcare,HIPAA,API"}],["$","meta","5",{"name":"robots","content":"index, follow"}],["$","meta","6",{"name":"googlebot","content":"index, follow"}],["$","link","7",{"rel":"canonical","href":"https://assistdocs.asimo.io"}],["$","meta","8",{"property":"og:title","content":"VoiceAssist Documentation"}],["$","meta","9",{"property":"og:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","10",{"property":"og:url","content":"https://assistdocs.asimo.io"}],["$","meta","11",{"property":"og:site_name","content":"VoiceAssist Docs"}],["$","meta","12",{"property":"og:type","content":"website"}],["$","meta","13",{"name":"twitter:card","content":"summary"}],["$","meta","14",{"name":"twitter:title","content":"VoiceAssist Documentation"}],["$","meta","15",{"name":"twitter:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","16",{"name":"next-size-adjust"}]] 1:null