Operations Overview

Last Updated: 2025-11-27

This document provides a central hub for all operations-related documentation for VoiceAssist.

Quick Links

Runbooks

All runbooks follow a standardized format with severity levels, step-by-step procedures, and verification steps.

Compliance

For HIPAA compliance, see Security & Compliance.

Incident Severity Levels

Key SLOs

On-Call Essentials

Quick Diagnostic Commands

# Check service health
curl http://localhost:8000/health
curl http://localhost:8000/ready

# Check all containers
docker compose ps

# View recent logs
docker compose logs --tail=100 voiceassist-server

# Check database
docker compose exec postgres psql -U voiceassist -c "SELECT 1"

# Check Redis
docker compose exec redis redis-cli ping

Escalation Path

1. L1 Support: Check health endpoints, restart services
2. L2 DevOps: Investigate logs, check metrics, apply standard fixes
3. L3 Engineering: Deep debugging, code-level investigation
4. Management: Major incidents requiring business decisions

Related Documentation

• Unified Architecture - System architecture
• Backend Architecture - Backend details
• Security & Compliance - HIPAA compliance
• Implementation Status - Component status

Version History

Deployment Runbook

Last Updated: 2025-11-27 Purpose: Step-by-step guide for deploying VoiceAssist V2

Pre-Deployment Checklist

• All tests passing in CI/CD
• Code reviewed and approved
• Database migrations reviewed
• Breaking changes documented
• Rollback plan documented
• Stakeholders notified
• Maintenance window scheduled (if required)

Deployment Steps

1. Pre-Deployment Verification

# Check current system health
curl http://localhost:8000/health
curl http://localhost:8000/ready

# Verify all containers running
docker compose ps

# Check database connection
docker compose exec postgres psql -U voiceassist -d voiceassist -c "SELECT version();"

# Check Redis
docker compose exec redis redis-cli ping

# Check Qdrant
curl http://localhost:6333/collections

2. Backup Current State

# Backup database
docker compose exec postgres pg_dump -U voiceassist voiceassist > backup_$(date +%Y%m%d_%H%M%S).sql

# Backup environment configuration
cp .env .env.backup_$(date +%Y%m%d_%H%M%S)

# Tag current Docker images
docker tag voiceassist-voiceassist-server:latest voiceassist-voiceassist-server:pre-deploy-$(date +%Y%m%d_%H%M%S)

3. Pull Latest Code

# Fetch latest changes
git fetch origin

# Check what's changing
git log --oneline HEAD..origin/main

# Pull changes
git pull origin main

# Verify correct branch
git branch --show-current
git log -1 --oneline

4. Update Environment Configuration

# Review .env changes
diff .env.example .env

# Update .env if needed
vim .env

# Validate configuration
grep -v '^#' .env | grep -v '^$' | wc -l  # Count non-empty lines

5. Run Database Migrations

# Check current migration status
docker compose run --rm voiceassist-server alembic current

# Review pending migrations
docker compose run --rm voiceassist-server alembic history

# Run migrations
docker compose run --rm voiceassist-server alembic upgrade head

# Verify migration success
docker compose run --rm voiceassist-server alembic current

6. Build New Images

# Build updated images
docker compose build voiceassist-server

# Verify image built
docker images | grep voiceassist-server

# Check image size (should be reasonable)
docker images voiceassist-voiceassist-server:latest --format "{{.Size}}"

7. Deploy Services

# Deploy with zero-downtime (recreate containers)
docker compose up -d voiceassist-server

# Watch logs for startup
docker compose logs -f voiceassist-server

# Wait for healthcheck
sleep 10

8. Post-Deployment Verification

# Check health endpoint
curl http://localhost:8000/health

# Check readiness
curl http://localhost:8000/ready

# Verify version
curl http://localhost:8000/health | jq '.version'

# Check all containers running
docker compose ps

# Check logs for errors
docker compose logs --tail=100 voiceassist-server | grep -i error

# Verify metrics endpoint
curl http://localhost:8000/metrics | head -20

# Test a sample API endpoint (requires auth)
# curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/users/me

9. Smoke Tests

# Test authentication
curl -X POST http://localhost:8000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"password"}' | jq '.'

# Test database connectivity
docker compose exec postgres psql -U voiceassist -d voiceassist -c "SELECT COUNT(*) FROM users;"

# Test Redis
docker compose exec redis redis-cli --raw incr deployment_test

# Test Qdrant
curl http://localhost:6333/collections

10. Monitor Initial Traffic

# Watch logs for first 5 minutes
docker compose logs -f --tail=100 voiceassist-server

# Monitor metrics
watch -n 5 'curl -s http://localhost:8000/metrics | grep -E "(http_requests_total|http_request_duration)"'

# Check error rate
docker compose logs --since 5m voiceassist-server | grep -i error | wc -l

Rollback Procedure

If deployment fails, follow these steps:

Quick Rollback (Image-Based)

# Stop current containers
docker compose down voiceassist-server

# Revert to previous image
PREVIOUS_TAG="pre-deploy-YYYYMMDD_HHMMSS"  # From backup step
docker tag voiceassist-voiceassist-server:$PREVIOUS_TAG voiceassist-voiceassist-server:latest

# Start previous version
docker compose up -d voiceassist-server

# Verify rollback
curl http://localhost:8000/health | jq '.version'

Full Rollback (Code + Database)

# Stop services
docker compose down voiceassist-server

# Revert code
git log -1 --oneline  # Note current commit
git checkout HEAD~1   # Or specific commit hash

# Rollback database migration
BACKUP_FILE="backup_YYYYMMDD_HHMMSS.sql"
docker compose exec -T postgres psql -U voiceassist voiceassist < $BACKUP_FILE

# Rebuild image
docker compose build voiceassist-server

# Start services
docker compose up -d voiceassist-server

# Verify rollback
curl http://localhost:8000/health

Deployment Checklist

Post-Deployment:

• Health endpoint returning 200
• Readiness endpoint returning 200
• No error logs in last 5 minutes
• Metrics endpoint accessible
• Database migrations applied
• All containers running
• Sample API requests successful
• Version number updated
• Stakeholders notified of completion
• Documentation updated (if needed)

Common Issues & Solutions

Issue: Database Migration Fails

Symptoms: Migration command returns error

Solution:

# Check current state
docker compose run --rm voiceassist-server alembic current

# Manually review SQL
docker compose run --rm voiceassist-server alembic show <revision>

# If safe, downgrade one step
docker compose run --rm voiceassist-server alembic downgrade -1

# Fix issue and retry
docker compose run --rm voiceassist-server alembic upgrade head

Issue: Container Won't Start

Symptoms: Container crashes immediately or fails healthcheck

Solution:

# Check logs
docker compose logs --tail=50 voiceassist-server

# Check container exit code
docker compose ps -a voiceassist-server

# Verify environment variables
docker compose config | grep -A 20 voiceassist-server

# Test dependencies
docker compose exec postgres pg_isready
docker compose exec redis redis-cli ping

Issue: High Error Rate After Deployment

Symptoms: Increased 5xx errors in logs/metrics

Solution:

# Check error logs
docker compose logs voiceassist-server | grep -i error

# Check database connections
docker compose exec postgres psql -U voiceassist -d voiceassist -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';"

# Check Redis memory
docker compose exec redis redis-cli INFO memory | grep used_memory_human

# Rollback if errors > 5% of traffic

Emergency Contacts

• On-Call Engineer: Check PagerDuty
• Database Admin: DBA on-call rotation
• DevOps Lead: ops-team@voiceassist.local
• Product Owner: product@voiceassist.local

##Related Documentation

• UNIFIED_ARCHITECTURE.md
• CONNECTION_POOL_OPTIMIZATION.md
• Incident Response Runbook
• Backup & Restore Runbook

Document Version: 1.0 Last Updated: 2025-11-21 Maintained By: VoiceAssist DevOps Team Review Cycle: After each major deployment or quarterly

Incident Response Runbook

Last Updated: 2025-11-27 Purpose: Comprehensive guide for handling incidents in VoiceAssist V2

Incident Severity Levels

Initial Response Procedure

1. Incident Detection

# Check system health
curl -s http://localhost:8000/health | jq '.'

# Expected output:
# {
#   "status": "healthy",
#   "version": "2.0.0",
#   "timestamp": "2025-11-21T..."
# }

# Check all services
docker compose ps

# Check recent error logs
docker compose logs --since 10m voiceassist-server | grep -i error

# Check metrics for anomalies
curl -s http://localhost:8000/metrics | grep -E "(error|failure)"

2. Immediate Triage (First 5 Minutes)

Checklist:

• Acknowledge the incident (update status page if available)
• Determine severity level using table above
• Notify on-call engineer if P1/P2
• Create incident tracking ticket/document
• Join incident response channel (Slack/Teams)

# Quick system overview
echo "=== System Status ==="
docker compose ps
echo ""
echo "=== Error Count (Last 10 min) ==="
docker compose logs --since 10m | grep -i error | wc -l
echo ""
echo "=== Active Database Connections ==="
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';"
echo ""
echo "=== Redis Memory ==="
docker compose exec redis redis-cli INFO memory | grep used_memory_human
echo ""
echo "=== Disk Usage ==="
df -h

3. Assess Impact

# Check request success rate
docker compose logs --since 15m voiceassist-server | \
  grep -oE "status=[0-9]+" | sort | uniq -c

# Check database connectivity
docker compose exec postgres pg_isready
docker compose exec postgres psql -U voiceassist -d voiceassist -c "SELECT 1;"

# Check Redis connectivity
docker compose exec redis redis-cli ping

# Check Qdrant connectivity
curl -s http://localhost:6333/healthz

# Check network connectivity
docker compose exec voiceassist-server ping -c 3 postgres
docker compose exec voiceassist-server ping -c 3 redis
docker compose exec voiceassist-server ping -c 3 qdrant

Incident Response by Severity

P1 - Critical Incident Response

Timeline: 0-15 minutes

1. Immediate Actions:

   • Page on-call engineer
   • Notify management
   • Update status page: "Investigating outage"
   • Join war room/incident call

2. Rapid Assessment:

# Check if complete outage
curl -s http://localhost:8000/health || echo "COMPLETE OUTAGE"

# Check all infrastructure
docker compose ps -a

# Check for recent deployments
git log -5 --oneline --since="2 hours ago"

# Check system resources
docker stats --no-stream

# Check disk space (common cause)
df -h
du -sh /var/lib/docker

3. Emergency Mitigation:

# Option 1: Restart all services
docker compose restart

# Option 2: Rollback recent deployment (if within 2 hours)
git log -1 --oneline  # Current version
git checkout HEAD~1   # Previous version
docker compose build voiceassist-server
docker compose up -d voiceassist-server

# Option 3: Scale up resources (if performance issue)
docker compose up -d --scale voiceassist-server=3

# Option 4: Enable maintenance mode
# Create maintenance mode flag
touch /tmp/maintenance_mode
docker compose exec voiceassist-server touch /app/maintenance_mode

4. Communication Template (P1):

Subject: [P1 INCIDENT] VoiceAssist Service Outage

Status: INVESTIGATING
Start Time: [TIME]
Impact: Complete service unavailable
Affected Users: All users
Incident Commander: [NAME]

Current Actions:
- Identified root cause as [X]
- Attempting mitigation via [Y]
- ETR: [TIME] (or "investigating")

Next Update: [TIME] (within 15 minutes)

P2 - High Severity Response

Timeline: 0-60 minutes

1. Assessment (First 15 minutes):

# Identify affected component
docker compose logs --since 30m voiceassist-server | grep -i error | tail -50

# Check specific service health
curl -s http://localhost:8000/ready | jq '.'

# Check database performance
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT pid, usename, application_name, state, query_start,
   wait_event_type, query
   FROM pg_stat_activity
   WHERE state != 'idle'
   ORDER BY query_start DESC
   LIMIT 20;"

# Check slow queries
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT query, calls, total_time, mean_time, max_time
   FROM pg_stat_statements
   ORDER BY mean_time DESC
   LIMIT 10;"

2. Mitigation Actions:

   • Isolate affected component
   • Enable fallback mechanisms
   • Scale affected service
   • Update monitoring thresholds

3. Communication Template (P2):

Subject: [P2 INCIDENT] VoiceAssist Degraded Performance

Status: MITIGATING
Start Time: [TIME]
Impact: [Specific feature] unavailable/degraded
Affected Users: [Percentage or specific user group]
Incident Commander: [NAME]

Timeline:
- [TIME]: Issue detected
- [TIME]: Root cause identified
- [TIME]: Mitigation in progress

Root Cause: [Brief description]
Mitigation: [Actions being taken]
ETR: [TIME]

Next Update: [TIME] (within 30 minutes)

P3 - Medium Severity Response

Timeline: 0-4 hours

1. Standard Investigation:

# Detailed log analysis
docker compose logs --since 1h voiceassist-server | grep -A 5 -B 5 "error"

# Check resource utilization trends
docker stats --no-stream

# Review recent changes
git log --since="24 hours ago" --oneline

# Check configuration
docker compose config | grep -A 10 voiceassist-server

2. Documented Fix Process:
   • Create issue in tracking system
   • Assign to appropriate team
   • Document reproduction steps
   • Implement fix
   • Test in staging (if available)
   • Deploy fix
   • Verify resolution

P4 - Low Severity Response

Standard ticket workflow - no immediate response required

Escalation Paths

When to Escalate

Escalate Immediately If:

• Unable to identify root cause within 30 minutes (P1) or 2 hours (P2)
• Mitigation attempts unsuccessful
• Data loss suspected
• Security breach suspected
• Multiple systems affected
• Customer data at risk

Escalation Chain

L1 - On-Call Engineer
  ↓ (30 min for P1, 2 hrs for P2)
L2 - Team Lead
  ↓ (1 hr for P1, 4 hrs for P2)
L3 - Engineering Manager
  ↓ (2 hrs for P1)
L4 - CTO / VP Engineering

Escalation Command Script

# Document current state before escalating
cat > /tmp/escalation_report_$(date +%Y%m%d_%H%M%S).txt <<EOF
ESCALATION REPORT
=================
Time: $(date)
Severity: P1/P2/P3/P4
Duration: [X hours]
Impact: [Description]

Current System State:
$(docker compose ps)

Recent Errors:
$(docker compose logs --since 30m voiceassist-server | grep -i error | tail -20)

Actions Attempted:
- [List all mitigation attempts]
- [Include results of each attempt]

Reason for Escalation:
[Clear explanation of why escalating]

Additional Context:
[Any other relevant information]
EOF

cat /tmp/escalation_report_$(date +%Y%m%d_%H%M%S).txt

Common Incident Types

Database Connection Issues

Symptoms:

• "Connection pool exhausted" errors
• "Too many connections" errors
• Slow response times

Investigation:

# Check connection pool status
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# Check max connections
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SHOW max_connections;"

# Check current connections
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT datname, usename, application_name, count(*)
   FROM pg_stat_activity
   GROUP BY datname, usename, application_name;"

# Kill idle connections
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state = 'idle'
   AND state_change < current_timestamp - INTERVAL '10 minutes';"

Resolution:

# Restart application to reset connection pool
docker compose restart voiceassist-server

# Temporarily increase connection pool
docker compose exec voiceassist-server sh -c \
  "export DB_POOL_SIZE=30 && supervisorctl restart all"

# Long-term: Update docker-compose.yml or .env
echo "DB_POOL_SIZE=30" >> .env
docker compose up -d voiceassist-server

Memory/Resource Exhaustion

Symptoms:

• Container restarts
• OOMKilled status
• Slow performance

Investigation:

# Check container memory usage
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Check for OOMKilled containers
docker inspect voiceassist-voiceassist-server-1 | grep OOMKilled

# Check system memory
free -h

# Check Redis memory
docker compose exec redis redis-cli INFO memory

Resolution:

# Increase memory limits in docker-compose.yml
# Edit docker-compose.yml to increase mem_limit

# Clear Redis cache if needed
docker compose exec redis redis-cli FLUSHDB

# Restart affected container
docker compose restart voiceassist-server

# Monitor memory after restart
watch -n 5 'docker stats --no-stream | grep voiceassist-server'

API Performance Degradation

Symptoms:

• Slow response times
• Timeout errors
• High request queue

Investigation:

# Check response times in metrics
curl -s http://localhost:8000/metrics | grep http_request_duration

# Check slow queries
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT pid, now() - query_start as duration, query
   FROM pg_stat_activity
   WHERE state != 'idle'
   AND now() - query_start > interval '5 seconds'
   ORDER BY duration DESC;"

# Check for locks
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT * FROM pg_locks WHERE NOT granted;"

# Check CPU usage
docker stats --no-stream

Resolution:

# Scale horizontally if needed
docker compose up -d --scale voiceassist-server=3

# Kill slow queries
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state != 'idle'
   AND now() - query_start > interval '30 seconds';"

# Enable query caching in Redis
docker compose exec redis redis-cli CONFIG SET maxmemory-policy allkeys-lru

Security Incidents

Symptoms:

• Unusual traffic patterns
• Unauthorized access attempts
• Data breach alerts

IMMEDIATE ACTIONS:

# 1. DO NOT DESTROY EVIDENCE
# 2. Document everything
# 3. Isolate affected systems

# Stop accepting new connections (if breach confirmed)
docker compose exec voiceassist-server iptables -A INPUT -p tcp --dport 8000 -j DROP

# Capture current state
docker compose logs > /tmp/security_incident_logs_$(date +%Y%m%d_%H%M%S).txt
docker compose exec postgres pg_dump -U voiceassist voiceassist > \
  /tmp/security_incident_db_$(date +%Y%m%d_%H%M%S).sql

# Check for suspicious activity
docker compose logs voiceassist-server | grep -E "401|403|429" | tail -100

# Check database for unauthorized access
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT * FROM user_sessions WHERE created_at > NOW() - INTERVAL '1 hour'
   ORDER BY created_at DESC;"

# Rotate credentials IMMEDIATELY
# Generate new secrets
openssl rand -base64 32 > /tmp/new_secret_key.txt

# Update .env with new credentials
# Force logout all users
docker compose exec redis redis-cli FLUSHALL

ESCALATION: Security incidents ALWAYS require immediate escalation to security team

Post-Incident Activities

Immediate Post-Incident (Within 1 Hour)

Checklist:

• Verify incident fully resolved
• Update status page to "Resolved"
• Send final communication to stakeholders
• Document timeline in incident ticket
• Schedule post-mortem meeting (within 48 hours for P1/P2)

# Verification script
echo "=== Post-Incident Verification ==="
echo "Health Check:"
curl -s http://localhost:8000/health | jq '.'
echo ""
echo "Error Rate (Last 30 min):"
docker compose logs --since 30m voiceassist-server | grep -i error | wc -l
echo ""
echo "Container Status:"
docker compose ps
echo ""
echo "Database Connections:"
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

Post-Mortem Process

Post-Mortem Template:

# Post-Mortem: [Incident Title]

## Incident Details

- **Date**: YYYY-MM-DD
- **Duration**: X hours Y minutes
- **Severity**: P1/P2/P3/P4
- **Incident Commander**: [Name]
- **Participants**: [Names]

## Impact

- **Users Affected**: [Number or percentage]
- **Services Affected**: [List]
- **Financial Impact**: [If applicable]
- **Data Loss**: None / [Description]

## Timeline

| Time  | Event                       |
| ----- | --------------------------- |
| HH:MM | Incident began              |
| HH:MM | Detected by [person/system] |
| HH:MM | Initial response started    |
| HH:MM | Root cause identified       |
| HH:MM | Mitigation deployed         |
| HH:MM | Incident resolved           |

## Root Cause

[Detailed explanation of what caused the incident]

## What Went Well

- [Things that worked during response]
- [Effective tools/processes]

## What Went Wrong

- [Issues encountered during response]
- [Gaps in tooling/process]

## Action Items

| Action                   | Owner  | Due Date | Priority |
| ------------------------ | ------ | -------- | -------- |
| [Preventive measure]     | [Name] | [Date]   | P1/P2/P3 |
| [Monitoring improvement] | [Name] | [Date]   | P1/P2/P3 |
| [Documentation update]   | [Name] | [Date]   | P1/P2/P3 |

## Lessons Learned

- [Key takeaway 1]
- [Key takeaway 2]
- [Key takeaway 3]

Post-Mortem Meeting Agenda

1. Review Timeline (10 minutes)

   • Walk through incident from detection to resolution
   • No blame, focus on facts

2. Root Cause Analysis (15 minutes)

   • Technical deep-dive
   • Use "5 Whys" technique

3. Impact Assessment (10 minutes)

   • User impact
   • Business impact
   • Reputation impact

4. Prevention Discussion (20 minutes)

   • How to prevent recurrence
   • Monitoring improvements
   • Process improvements

5. Action Items (5 minutes)

   • Assign owners and due dates
   • Set follow-up meeting

Communication Templates

Initial Notification (P1/P2)

Subject: [P1/P2] VoiceAssist Service Issue - [Brief Description]

Dear Team,

We are currently experiencing [issue description] affecting [scope of impact].

Status: INVESTIGATING
Start Time: [TIME]
Severity: P1/P2
Impact: [Description]
Affected Systems: [List]
Incident Commander: [NAME]

We are actively working to resolve this issue and will provide updates
every [15 minutes for P1, 30 minutes for P2].

Next Update: [TIME]

VoiceAssist Operations Team

Status Update (During Incident)

Subject: [UPDATE - P1/P2] VoiceAssist Service Issue - [Brief Description]

Update #[N] - [TIME]

Current Status: [INVESTIGATING/IDENTIFIED/MITIGATING/RESOLVED]

Progress:
- [What we've learned]
- [What we've tried]
- [Current approach]

Impact Update: [Any changes to scope]

Next Steps:
- [Action 1]
- [Action 2]

ETR: [Estimated Time to Resolution or "investigating"]

Next Update: [TIME]

VoiceAssist Operations Team

Resolution Notification

Subject: [RESOLVED - P1/P2] VoiceAssist Service Issue - [Brief Description]

Status: RESOLVED
Resolution Time: [TIME]
Total Duration: [X hours Y minutes]

The issue affecting [description] has been fully resolved.

Root Cause: [Brief explanation]

Resolution: [What was done to fix it]

Impact Summary:
- Users Affected: [Number/Percentage]
- Duration: [X hours Y minutes]
- Data Loss: None / [Description]

Next Steps:
- Post-mortem scheduled for [DATE/TIME]
- Preventive measures being implemented

We apologize for any inconvenience this may have caused.

VoiceAssist Operations Team

Incident Response Tools

Quick Command Reference

# Health Check Bundle
alias va-health='curl -s http://localhost:8000/health | jq .'
alias va-ready='curl -s http://localhost:8000/ready | jq .'
alias va-metrics='curl -s http://localhost:8000/metrics'

# Log Analysis
alias va-errors='docker compose logs --since 10m voiceassist-server | grep -i error'
alias va-errors-count='docker compose logs --since 10m voiceassist-server | grep -i error | wc -l'
alias va-logs-tail='docker compose logs -f --tail=100 voiceassist-server'

# Resource Check
alias va-stats='docker stats --no-stream | grep voiceassist'
alias va-disk='df -h | grep -E "(Filesystem|/dev/)"'

# Database Quick Checks
alias va-db-connections='docker compose exec postgres psql -U voiceassist -d voiceassist -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"'
alias va-db-slow='docker compose exec postgres psql -U voiceassist -d voiceassist -c "SELECT pid, now() - query_start as duration, query FROM pg_stat_activity WHERE state != '\''idle'\'' ORDER BY duration DESC LIMIT 10;"'

# Redis Checks
alias va-redis-info='docker compose exec redis redis-cli INFO'
alias va-redis-memory='docker compose exec redis redis-cli INFO memory | grep used_memory_human'

Incident Response Script

#!/bin/bash
# Save as: /usr/local/bin/va-incident-check

echo "=== VoiceAssist Incident Response Check ==="
echo "Time: $(date)"
echo ""

echo "=== 1. Service Health ==="
curl -s http://localhost:8000/health | jq '.' || echo "HEALTH CHECK FAILED"
echo ""

echo "=== 2. Container Status ==="
docker compose ps
echo ""

echo "=== 3. Recent Errors (Last 10 min) ==="
ERROR_COUNT=$(docker compose logs --since 10m voiceassist-server 2>/dev/null | grep -i error | wc -l)
echo "Error Count: $ERROR_COUNT"
if [ "$ERROR_COUNT" -gt 10 ]; then
    echo "⚠️  HIGH ERROR RATE DETECTED"
    docker compose logs --since 10m voiceassist-server | grep -i error | tail -10
fi
echo ""

echo "=== 4. Database Status ==="
docker compose exec -T postgres pg_isready || echo "DATABASE NOT READY"
docker compose exec -T postgres psql -U voiceassist -d voiceassist -c \
  "SELECT count(*), state FROM pg_stat_activity GROUP BY state;" 2>/dev/null
echo ""

echo "=== 5. Redis Status ==="
docker compose exec -T redis redis-cli ping || echo "REDIS NOT RESPONDING"
docker compose exec -T redis redis-cli INFO memory | grep used_memory_human
echo ""

echo "=== 6. Resource Usage ==="
docker stats --no-stream | grep voiceassist
echo ""

echo "=== 7. Disk Space ==="
df -h | grep -E "(Filesystem|/$|/var)"
echo ""

echo "=== Summary ==="
if [ "$ERROR_COUNT" -gt 50 ]; then
    echo "🔴 CRITICAL - High error rate detected"
elif [ "$ERROR_COUNT" -gt 10 ]; then
    echo "🟡 WARNING - Elevated error rate"
else
    echo "🟢 OK - System appears healthy"
fi

Emergency Contacts

Primary Contacts

Escalation Contacts

External Contacts

• Cloud Provider Support: [Support portal URL]
• Third-party Services: [Service provider contacts]
• Legal (for security incidents): legal@voiceassist.local

Related Documentation

• Deployment Runbook
• Backup & Restore Runbook
• Monitoring Runbook
• Troubleshooting Runbook
• Scaling Runbook
• UNIFIED_ARCHITECTURE.md
• CONNECTION_POOL_OPTIMIZATION.md

Document Version: 1.0 Last Updated: 2025-11-21 Maintained By: VoiceAssist DevOps Team Review Cycle: Monthly or after each P1/P2 incident Next Review: 2025-12-21

Backup & Restore Runbook

Last Updated: 2025-11-27 Purpose: Comprehensive guide for backup and restore operations in VoiceAssist V2

Backup Strategy Overview

Backup Schedule

Backup Storage Locations

# Default backup directory structure
/backups/
├── postgres/
│   ├── daily/
│   ├── weekly/
│   └── monthly/
├── redis/
├── qdrant/
├── config/
├── volumes/
└── logs/

PostgreSQL Database Backup

Full Database Backup

# Create timestamped backup
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/postgres/daily"

# Ensure backup directory exists
mkdir -p $BACKUP_DIR

# Full database dump
docker compose exec -T postgres pg_dump \
  -U voiceassist \
  -d voiceassist \
  -F c \
  -b \
  -v \
  -f /tmp/voiceassist_${BACKUP_DATE}.dump

# Copy from container to host
docker compose cp postgres:/tmp/voiceassist_${BACKUP_DATE}.dump \
  ${BACKUP_DIR}/voiceassist_${BACKUP_DATE}.dump

# Verify backup
ls -lh ${BACKUP_DIR}/voiceassist_${BACKUP_DATE}.dump

# Expected output: File size should be > 0 bytes

Compressed SQL Backup

# SQL format with compression
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/postgres/daily"

mkdir -p $BACKUP_DIR

# Create compressed SQL dump
docker compose exec -T postgres pg_dump \
  -U voiceassist \
  -d voiceassist \
  --clean \
  --if-exists \
  --verbose \
  | gzip > ${BACKUP_DIR}/voiceassist_${BACKUP_DATE}.sql.gz

# Verify backup
ls -lh ${BACKUP_DIR}/voiceassist_${BACKUP_DATE}.sql.gz
gunzip -t ${BACKUP_DIR}/voiceassist_${BACKUP_DATE}.sql.gz && echo "✓ Backup file is valid"

Schema-Only Backup

# Backup schema structure only (useful for development)
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)

docker compose exec -T postgres pg_dump \
  -U voiceassist \
  -d voiceassist \
  --schema-only \
  --no-owner \
  --no-acl \
  > /backups/postgres/schema_${BACKUP_DATE}.sql

echo "Schema backup completed: schema_${BACKUP_DATE}.sql"

Table-Specific Backup

# Backup specific tables
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
TABLES="users conversations messages"

for TABLE in $TABLES; do
  echo "Backing up table: $TABLE"
  docker compose exec -T postgres pg_dump \
    -U voiceassist \
    -d voiceassist \
    -t $TABLE \
    --data-only \
    | gzip > /backups/postgres/table_${TABLE}_${BACKUP_DATE}.sql.gz
done

echo "Table backups completed"

Automated Backup Script

#!/bin/bash
# Save as: /usr/local/bin/va-backup-postgres

set -e

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/postgres"
DAILY_DIR="${BACKUP_DIR}/daily"
WEEKLY_DIR="${BACKUP_DIR}/weekly"
MONTHLY_DIR="${BACKUP_DIR}/monthly"
LOG_FILE="${BACKUP_DIR}/backup.log"

# Ensure directories exist
mkdir -p $DAILY_DIR $WEEKLY_DIR $MONTHLY_DIR

# Function to log messages
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a $LOG_FILE
}

log "Starting PostgreSQL backup"

# Daily backup
log "Creating daily backup"
docker compose exec -T postgres pg_dump \
  -U voiceassist \
  -d voiceassist \
  -F c \
  -b \
  | gzip > ${DAILY_DIR}/voiceassist_${BACKUP_DATE}.dump.gz

if [ $? -eq 0 ]; then
    log "Daily backup completed: ${BACKUP_DATE}.dump.gz"
    BACKUP_SIZE=$(du -h ${DAILY_DIR}/voiceassist_${BACKUP_DATE}.dump.gz | cut -f1)
    log "Backup size: ${BACKUP_SIZE}"
else
    log "ERROR: Daily backup failed"
    exit 1
fi

# Weekly backup (every Sunday)
if [ $(date +%u) -eq 7 ]; then
    log "Creating weekly backup"
    cp ${DAILY_DIR}/voiceassist_${BACKUP_DATE}.dump.gz \
       ${WEEKLY_DIR}/voiceassist_week_$(date +%Y%U).dump.gz
    log "Weekly backup created"
fi

# Monthly backup (first day of month)
if [ $(date +%d) -eq 01 ]; then
    log "Creating monthly backup"
    cp ${DAILY_DIR}/voiceassist_${BACKUP_DATE}.dump.gz \
       ${MONTHLY_DIR}/voiceassist_$(date +%Y%m).dump.gz
    log "Monthly backup created"
fi

# Cleanup old daily backups (keep 30 days)
log "Cleaning up old daily backups"
find ${DAILY_DIR} -name "voiceassist_*.dump.gz" -mtime +30 -delete

# Cleanup old weekly backups (keep 12 weeks)
find ${WEEKLY_DIR} -name "voiceassist_week_*.dump.gz" -mtime +84 -delete

# Cleanup old monthly backups (keep 12 months)
find ${MONTHLY_DIR} -name "voiceassist_*.dump.gz" -mtime +365 -delete

log "Backup process completed successfully"

Backup Verification

# Verify backup integrity
BACKUP_FILE="/backups/postgres/daily/voiceassist_20251121_120000.dump.gz"

# Check file exists and size
if [ -f "$BACKUP_FILE" ]; then
    echo "✓ Backup file exists"
    ls -lh $BACKUP_FILE
else
    echo "✗ Backup file not found"
    exit 1
fi

# Test extraction
gunzip -t $BACKUP_FILE
if [ $? -eq 0 ]; then
    echo "✓ Backup file is not corrupted"
else
    echo "✗ Backup file is corrupted"
    exit 1
fi

# Test restore to temporary database (recommended)
echo "Testing restore to temporary database..."
docker compose exec -T postgres psql -U voiceassist -c "CREATE DATABASE test_restore;"
gunzip -c $BACKUP_FILE | docker compose exec -T postgres pg_restore \
  -U voiceassist \
  -d test_restore \
  --verbose

if [ $? -eq 0 ]; then
    echo "✓ Backup restore test successful"
    docker compose exec -T postgres psql -U voiceassist -c "DROP DATABASE test_restore;"
else
    echo "✗ Backup restore test failed"
    docker compose exec -T postgres psql -U voiceassist -c "DROP DATABASE IF EXISTS test_restore;"
    exit 1
fi

PostgreSQL Database Restore

Pre-Restore Checklist

• Verify backup file integrity
• Ensure sufficient disk space
• Notify all users of maintenance
• Stop application services
• Create a backup of current database (before restore)
• Document current state

Full Database Restore

# Stop application to prevent connections
docker compose stop voiceassist-server

# Verify no active connections
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT count(*) FROM pg_stat_activity WHERE datname = 'voiceassist' AND pid != pg_backend_pid();"

# Terminate active connections if any
docker compose exec postgres psql -U voiceassist -d postgres -c \
  "SELECT pg_terminate_backend(pid) FROM pg_stat_activity
   WHERE datname = 'voiceassist' AND pid != pg_backend_pid();"

# Drop and recreate database
docker compose exec postgres psql -U voiceassist -d postgres <<EOF
DROP DATABASE IF EXISTS voiceassist;
CREATE DATABASE voiceassist OWNER voiceassist;
EOF

# Restore from custom format dump
BACKUP_FILE="/backups/postgres/daily/voiceassist_20251121_120000.dump.gz"

gunzip -c $BACKUP_FILE | docker compose exec -T postgres pg_restore \
  -U voiceassist \
  -d voiceassist \
  --verbose \
  --no-owner \
  --no-acl

# Verify restore
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT schemaname, tablename FROM pg_tables WHERE schemaname = 'public';"

# Restart application
docker compose start voiceassist-server

echo "Database restore completed"

Restore from SQL Dump

# For plain SQL dumps
BACKUP_FILE="/backups/postgres/daily/voiceassist_20251121_120000.sql.gz"

# Stop application
docker compose stop voiceassist-server

# Restore SQL
gunzip -c $BACKUP_FILE | docker compose exec -T postgres psql \
  -U voiceassist \
  -d voiceassist

# Restart application
docker compose start voiceassist-server

Point-in-Time Recovery (PITR)

# Requires WAL archiving to be enabled in PostgreSQL configuration

# 1. Stop database
docker compose stop postgres

# 2. Replace data directory with base backup
BACKUP_DIR="/backups/postgres/base"
DATA_DIR="/var/lib/docker/volumes/voiceassist_postgres_data/_data"

# Backup current data
mv $DATA_DIR ${DATA_DIR}.backup_$(date +%Y%m%d_%H%M%S)

# Restore base backup
cp -r $BACKUP_DIR $DATA_DIR

# 3. Create recovery configuration
cat > ${DATA_DIR}/recovery.conf <<EOF
restore_command = 'cp /backups/postgres/wal_archive/%f %p'
recovery_target_time = '2025-11-21 12:00:00'
EOF

# 4. Start PostgreSQL (will perform recovery)
docker compose start postgres

# 5. Monitor recovery
docker compose logs -f postgres | grep -i recovery

Partial Restore (Single Table)

# Restore specific table from backup
TABLE_NAME="users"
BACKUP_FILE="/backups/postgres/table_users_20251121_120000.sql.gz"

# Drop existing table data
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "TRUNCATE TABLE ${TABLE_NAME} CASCADE;"

# Restore table
gunzip -c $BACKUP_FILE | docker compose exec -T postgres psql \
  -U voiceassist \
  -d voiceassist

# Verify
docker compose exec postgres psql -U voiceassist -d voiceassist -c \
  "SELECT COUNT(*) FROM ${TABLE_NAME};"

Redis Backup

Manual Redis Backup

# Trigger Redis save
docker compose exec redis redis-cli BGSAVE

# Wait for save to complete
docker compose exec redis redis-cli LASTSAVE

# Copy RDB file
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p /backups/redis

docker compose cp redis:/data/dump.rdb \
  /backups/redis/dump_${BACKUP_DATE}.rdb

# Verify backup
ls -lh /backups/redis/dump_${BACKUP_DATE}.rdb

Automated Redis Backup Script

#!/bin/bash
# Save as: /usr/local/bin/va-backup-redis

set -e

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/redis"
LOG_FILE="${BACKUP_DIR}/backup.log"

mkdir -p $BACKUP_DIR

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a $LOG_FILE
}

log "Starting Redis backup"

# Trigger background save
docker compose exec -T redis redis-cli BGSAVE > /dev/null

# Wait for save to complete (check every 2 seconds)
TIMEOUT=60
ELAPSED=0
while [ $ELAPSED -lt $TIMEOUT ]; do
    STATUS=$(docker compose exec -T redis redis-cli LASTSAVE 2>/dev/null || echo "0")
    if [ ! -z "$STATUS" ]; then
        break
    fi
    sleep 2
    ELAPSED=$((ELAPSED + 2))
done

# Copy RDB file
docker compose cp redis:/data/dump.rdb \
  ${BACKUP_DIR}/dump_${BACKUP_DATE}.rdb

if [ $? -eq 0 ]; then
    log "Redis backup completed: dump_${BACKUP_DATE}.rdb"
    BACKUP_SIZE=$(du -h ${BACKUP_DIR}/dump_${BACKUP_DATE}.rdb | cut -f1)
    log "Backup size: ${BACKUP_SIZE}"
else
    log "ERROR: Redis backup failed"
    exit 1
fi

# Cleanup old backups (keep 7 days)
find ${BACKUP_DIR} -name "dump_*.rdb" -mtime +7 -delete
log "Cleanup completed"

log "Redis backup process completed successfully"

Redis Restore

# Stop Redis
docker compose stop redis

# Replace RDB file
BACKUP_FILE="/backups/redis/dump_20251121_120000.rdb"

docker compose cp $BACKUP_FILE redis:/data/dump.rdb

# Start Redis (will load from dump.rdb)
docker compose start redis

# Verify data loaded
docker compose exec redis redis-cli DBSIZE

echo "Redis restore completed"

Qdrant Vector Database Backup

Create Qdrant Snapshot

# Create snapshot for specific collection
COLLECTION_NAME="voice_embeddings"
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/qdrant"

mkdir -p $BACKUP_DIR

# Create snapshot via API
SNAPSHOT_NAME=$(curl -X POST \
  "http://localhost:6333/collections/${COLLECTION_NAME}/snapshots" \
  | jq -r '.result.name')

echo "Snapshot created: $SNAPSHOT_NAME"

# Download snapshot
curl -X GET \
  "http://localhost:6333/collections/${COLLECTION_NAME}/snapshots/${SNAPSHOT_NAME}" \
  -o ${BACKUP_DIR}/${COLLECTION_NAME}_${BACKUP_DATE}.snapshot

# Verify backup
ls -lh ${BACKUP_DIR}/${COLLECTION_NAME}_${BACKUP_DATE}.snapshot

Backup All Qdrant Collections

#!/bin/bash
# Save as: /usr/local/bin/va-backup-qdrant

set -e

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/qdrant"
LOG_FILE="${BACKUP_DIR}/backup.log"

mkdir -p $BACKUP_DIR

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a $LOG_FILE
}

log "Starting Qdrant backup"

# Get all collections
COLLECTIONS=$(curl -s http://localhost:6333/collections | jq -r '.result.collections[].name')

if [ -z "$COLLECTIONS" ]; then
    log "No collections found"
    exit 0
fi

# Backup each collection
for COLLECTION in $COLLECTIONS; do
    log "Backing up collection: $COLLECTION"

    # Create snapshot
    SNAPSHOT_NAME=$(curl -s -X POST \
      "http://localhost:6333/collections/${COLLECTION}/snapshots" \
      | jq -r '.result.name')

    if [ ! -z "$SNAPSHOT_NAME" ] && [ "$SNAPSHOT_NAME" != "null" ]; then
        # Download snapshot
        curl -s -X GET \
          "http://localhost:6333/collections/${COLLECTION}/snapshots/${SNAPSHOT_NAME}" \
          -o ${BACKUP_DIR}/${COLLECTION}_${BACKUP_DATE}.snapshot

        log "Backup completed: ${COLLECTION}_${BACKUP_DATE}.snapshot"
        BACKUP_SIZE=$(du -h ${BACKUP_DIR}/${COLLECTION}_${BACKUP_DATE}.snapshot | cut -f1)
        log "Backup size: ${BACKUP_SIZE}"

        # Delete remote snapshot to save space
        curl -s -X DELETE \
          "http://localhost:6333/collections/${COLLECTION}/snapshots/${SNAPSHOT_NAME}" \
          > /dev/null
    else
        log "ERROR: Failed to create snapshot for $COLLECTION"
    fi
done

# Cleanup old backups (keep 14 days)
find ${BACKUP_DIR} -name "*.snapshot" -mtime +14 -delete
log "Cleanup completed"

log "Qdrant backup process completed successfully"

Qdrant Restore

# Stop Qdrant
docker compose stop qdrant

# Clear existing data (optional, for full restore)
docker compose exec qdrant rm -rf /qdrant/storage/*

# Start Qdrant
docker compose start qdrant

# Wait for Qdrant to be ready
sleep 5

# Restore each collection
COLLECTION_NAME="voice_embeddings"
BACKUP_FILE="/backups/qdrant/voice_embeddings_20251121_120000.snapshot"

# Upload snapshot
curl -X POST \
  "http://localhost:6333/collections/${COLLECTION_NAME}/snapshots/upload" \
  -H "Content-Type: multipart/form-data" \
  -F "snapshot=@${BACKUP_FILE}"

# Verify collection restored
curl -s http://localhost:6333/collections/${COLLECTION_NAME} | jq '.result'

echo "Qdrant restore completed"

Configuration Files Backup

Backup Configuration

#!/bin/bash
# Save as: /usr/local/bin/va-backup-config

set -e

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/config"
PROJECT_DIR="/Users/mohammednazmy/VoiceAssist"

mkdir -p $BACKUP_DIR

echo "Starting configuration backup"

# Create tarball of configuration files
tar -czf ${BACKUP_DIR}/config_${BACKUP_DATE}.tar.gz \
  -C $PROJECT_DIR \
  .env \
  docker-compose.yml \
  docker-compose.override.yml \
  alembic.ini \
  pyproject.toml \
  --exclude='.git' \
  --exclude='__pycache__'

# Encrypt backup (recommended for sensitive configs)
if command -v gpg &> /dev/null; then
    gpg --symmetric --cipher-algo AES256 \
      -o ${BACKUP_DIR}/config_${BACKUP_DATE}.tar.gz.gpg \
      ${BACKUP_DIR}/config_${BACKUP_DATE}.tar.gz

    # Remove unencrypted version
    rm ${BACKUP_DIR}/config_${BACKUP_DATE}.tar.gz
    echo "Configuration backup encrypted: config_${BACKUP_DATE}.tar.gz.gpg"
else
    echo "Configuration backup created: config_${BACKUP_DATE}.tar.gz"
    echo "WARNING: Backup is not encrypted. Consider installing gpg."
fi

# Cleanup old backups (keep 90 days)
find ${BACKUP_DIR} -name "config_*.tar.gz*" -mtime +90 -delete

echo "Configuration backup completed"

Restore Configuration

# For encrypted backups
BACKUP_FILE="/backups/config/config_20251121_120000.tar.gz.gpg"
PROJECT_DIR="/Users/mohammednazmy/VoiceAssist"

# Decrypt and extract
gpg --decrypt $BACKUP_FILE | tar -xzf - -C $PROJECT_DIR

# For unencrypted backups
BACKUP_FILE="/backups/config/config_20251121_120000.tar.gz"
tar -xzf $BACKUP_FILE -C $PROJECT_DIR

echo "Configuration restored"

Docker Volumes Backup

Backup Docker Volumes

#!/bin/bash
# Save as: /usr/local/bin/va-backup-volumes

set -e

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/volumes"

mkdir -p $BACKUP_DIR

echo "Starting Docker volumes backup"

# List of volumes to backup
VOLUMES=(
  "voiceassist_postgres_data"
  "voiceassist_redis_data"
  "voiceassist_qdrant_storage"
)

for VOLUME in "${VOLUMES[@]}"; do
    echo "Backing up volume: $VOLUME"

    # Create tarball of volume
    docker run --rm \
      -v ${VOLUME}:/source:ro \
      -v ${BACKUP_DIR}:/backup \
      alpine \
      tar -czf /backup/${VOLUME}_${BACKUP_DATE}.tar.gz -C /source .

    if [ $? -eq 0 ]; then
        echo "Backup completed: ${VOLUME}_${BACKUP_DATE}.tar.gz"
        BACKUP_SIZE=$(du -h ${BACKUP_DIR}/${VOLUME}_${BACKUP_DATE}.tar.gz | cut -f1)
        echo "Backup size: ${BACKUP_SIZE}"
    else
        echo "ERROR: Backup failed for $VOLUME"
    fi
done

# Cleanup old backups (keep 30 days)
find ${BACKUP_DIR} -name "*.tar.gz" -mtime +30 -delete

echo "Docker volumes backup completed"

Restore Docker Volumes

# Stop services
docker compose down

# Restore specific volume
VOLUME_NAME="voiceassist_postgres_data"
BACKUP_FILE="/backups/volumes/voiceassist_postgres_data_20251121_120000.tar.gz"

# Remove existing volume (WARNING: destructive)
docker volume rm $VOLUME_NAME

# Create new volume
docker volume create $VOLUME_NAME

# Restore data
docker run --rm \
  -v ${VOLUME_NAME}:/target \
  -v $(dirname $BACKUP_FILE):/backup \
  alpine \
  tar -xzf /backup/$(basename $BACKUP_FILE) -C /target

echo "Volume $VOLUME_NAME restored"

# Start services
docker compose up -d

Disaster Recovery

Complete System Backup

#!/bin/bash
# Save as: /usr/local/bin/va-backup-full

set -e

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_ROOT="/backups"
DR_DIR="${BACKUP_ROOT}/disaster_recovery"

mkdir -p $DR_DIR

echo "============================================"
echo "Starting Full System Backup for DR"
echo "Date: $(date)"
echo "============================================"

# Stop application (keep databases running)
docker compose stop voiceassist-server

# 1. Backup PostgreSQL
echo "[1/5] Backing up PostgreSQL..."
/usr/local/bin/va-backup-postgres

# 2. Backup Redis
echo "[2/5] Backing up Redis..."
/usr/local/bin/va-backup-redis

# 3. Backup Qdrant
echo "[3/5] Backing up Qdrant..."
/usr/local/bin/va-backup-qdrant

# 4. Backup Configuration
echo "[4/5] Backing up Configuration..."
/usr/local/bin/va-backup-config

# 5. Backup Docker Volumes
echo "[5/5] Backing up Docker Volumes..."
/usr/local/bin/va-backup-volumes

# Create DR manifest
cat > ${DR_DIR}/manifest_${BACKUP_DATE}.txt <<EOF
VoiceAssist V2 Disaster Recovery Backup
========================================
Date: $(date)
Backup ID: ${BACKUP_DATE}

Components Backed Up:
- PostgreSQL Database
- Redis Cache
- Qdrant Vector Database
- Configuration Files
- Docker Volumes

Backup Locations:
- PostgreSQL: ${BACKUP_ROOT}/postgres/daily/
- Redis: ${BACKUP_ROOT}/redis/
- Qdrant: ${BACKUP_ROOT}/qdrant/
- Config: ${BACKUP_ROOT}/config/
- Volumes: ${BACKUP_ROOT}/volumes/

Backup Sizes:
$(du -sh ${BACKUP_ROOT}/postgres/daily/voiceassist_${BACKUP_DATE}* 2>/dev/null || echo "PostgreSQL: N/A")
$(du -sh ${BACKUP_ROOT}/redis/dump_${BACKUP_DATE}.rdb 2>/dev/null || echo "Redis: N/A")
$(du -sh ${BACKUP_ROOT}/qdrant/*_${BACKUP_DATE}.snapshot 2>/dev/null || echo "Qdrant: N/A")
$(du -sh ${BACKUP_ROOT}/config/config_${BACKUP_DATE}.tar.gz* 2>/dev/null || echo "Config: N/A")

Total Backup Size:
$(du -sh ${BACKUP_ROOT} | cut -f1)

Verification Status:
- PostgreSQL: $(test -f ${BACKUP_ROOT}/postgres/daily/voiceassist_${BACKUP_DATE}* && echo "✓" || echo "✗")
- Redis: $(test -f ${BACKUP_ROOT}/redis/dump_${BACKUP_DATE}.rdb && echo "✓" || echo "✗")
- Config: $(test -f ${BACKUP_ROOT}/config/config_${BACKUP_DATE}.tar.gz* && echo "✓" || echo "✗")

Restore Command:
/usr/local/bin/va-restore-full ${BACKUP_DATE}
EOF

# Create compressed archive of entire backup
echo "Creating DR archive..."
tar -czf ${DR_DIR}/voiceassist_dr_${BACKUP_DATE}.tar.gz \
  -C ${BACKUP_ROOT} \
  postgres/daily \
  redis \
  qdrant \
  config \
  volumes

# Restart application
docker compose start voiceassist-server

echo "============================================"
echo "Full System Backup Completed"
echo "Manifest: ${DR_DIR}/manifest_${BACKUP_DATE}.txt"
echo "Archive: ${DR_DIR}/voiceassist_dr_${BACKUP_DATE}.tar.gz"
echo "============================================"

cat ${DR_DIR}/manifest_${BACKUP_DATE}.txt

Complete System Restore

#!/bin/bash
# Save as: /usr/local/bin/va-restore-full

set -e

if [ -z "$1" ]; then
    echo "Usage: $0 <backup_date>"
    echo "Example: $0 20251121_120000"
    exit 1
fi

BACKUP_DATE=$1
BACKUP_ROOT="/backups"

echo "============================================"
echo "Starting Full System Restore"
echo "Backup Date: ${BACKUP_DATE}"
echo "============================================"

# Verify manifest exists
MANIFEST="${BACKUP_ROOT}/disaster_recovery/manifest_${BACKUP_DATE}.txt"
if [ ! -f "$MANIFEST" ]; then
    echo "ERROR: Manifest not found: $MANIFEST"
    exit 1
fi

echo "Manifest found. Displaying backup details:"
cat $MANIFEST
echo ""

read -p "Do you want to proceed with restore? This will OVERWRITE all data (yes/no): " CONFIRM
if [ "$CONFIRM" != "yes" ]; then
    echo "Restore cancelled"
    exit 0
fi

# Stop all services
echo "Stopping services..."
docker compose down

# 1. Restore PostgreSQL
echo "[1/5] Restoring PostgreSQL..."
POSTGRES_BACKUP="${BACKUP_ROOT}/postgres/daily/voiceassist_${BACKUP_DATE}.dump.gz"
if [ -f "$POSTGRES_BACKUP" ]; then
    docker compose up -d postgres
    sleep 10

    docker compose exec postgres psql -U voiceassist -d postgres -c \
      "DROP DATABASE IF EXISTS voiceassist;"
    docker compose exec postgres psql -U voiceassist -d postgres -c \
      "CREATE DATABASE voiceassist OWNER voiceassist;"

    gunzip -c $POSTGRES_BACKUP | docker compose exec -T postgres pg_restore \
      -U voiceassist \
      -d voiceassist \
      --verbose \
      --no-owner \
      --no-acl

    echo "✓ PostgreSQL restored"
else
    echo "✗ PostgreSQL backup not found"
fi

# 2. Restore Redis
echo "[2/5] Restoring Redis..."
REDIS_BACKUP="${BACKUP_ROOT}/redis/dump_${BACKUP_DATE}.rdb"
if [ -f "$REDIS_BACKUP" ]; then
    docker compose stop redis
    docker compose cp $REDIS_BACKUP redis:/data/dump.rdb
    docker compose start redis
    sleep 5
    echo "✓ Redis restored"
else
    echo "✗ Redis backup not found"
fi

# 3. Restore Qdrant
echo "[3/5] Restoring Qdrant..."
docker compose up -d qdrant
sleep 10

for SNAPSHOT in ${BACKUP_ROOT}/qdrant/*_${BACKUP_DATE}.snapshot; do
    if [ -f "$SNAPSHOT" ]; then
        COLLECTION=$(basename $SNAPSHOT | sed "s/_${BACKUP_DATE}.snapshot//")
        echo "Restoring collection: $COLLECTION"

        curl -X POST \
          "http://localhost:6333/collections/${COLLECTION}/snapshots/upload" \
          -H "Content-Type: multipart/form-data" \
          -F "snapshot=@${SNAPSHOT}"

        echo "✓ Collection $COLLECTION restored"
    fi
done

# 4. Restore Configuration
echo "[4/5] Restoring Configuration..."
CONFIG_BACKUP="${BACKUP_ROOT}/config/config_${BACKUP_DATE}.tar.gz"
CONFIG_BACKUP_ENC="${CONFIG_BACKUP}.gpg"

if [ -f "$CONFIG_BACKUP_ENC" ]; then
    gpg --decrypt $CONFIG_BACKUP_ENC | tar -xzf - -C /Users/mohammednazmy/VoiceAssist
    echo "✓ Configuration restored (encrypted)"
elif [ -f "$CONFIG_BACKUP" ]; then
    tar -xzf $CONFIG_BACKUP -C /Users/mohammednazmy/VoiceAssist
    echo "✓ Configuration restored"
else
    echo "✗ Configuration backup not found"
fi

# 5. Start all services
echo "[5/5] Starting all services..."
docker compose up -d

# Wait for services to be ready
echo "Waiting for services to be ready..."
sleep 30

# Verify system health
echo ""
echo "============================================"
echo "Restore Completed - Verifying System Health"
echo "============================================"

curl -s http://localhost:8000/health | jq '.'
docker compose ps

echo ""
echo "Full system restore completed"
echo "Please verify all functionality before resuming operations"

Disaster Recovery Scenarios

Scenario 1: Complete Hardware Failure

# On NEW hardware:

# 1. Install Docker and Docker Compose
# 2. Clone repository
git clone <repository_url> /Users/mohammednazmy/VoiceAssist
cd /Users/mohammednazmy/VoiceAssist

# 3. Copy DR archive from backup location
scp backup-server:/backups/disaster_recovery/voiceassist_dr_YYYYMMDD_HHMMSS.tar.gz /tmp/

# 4. Extract DR archive
mkdir -p /backups
tar -xzf /tmp/voiceassist_dr_YYYYMMDD_HHMMSS.tar.gz -C /backups

# 5. Run full restore
/usr/local/bin/va-restore-full YYYYMMDD_HHMMSS

# 6. Verify and resume operations

Scenario 2: Data Corruption

# 1. Stop application
docker compose stop voiceassist-server

# 2. Create backup of corrupted data (for analysis)
/usr/local/bin/va-backup-full

# 3. Identify last known good backup
ls -lh /backups/disaster_recovery/manifest_*.txt

# 4. Restore from last good backup
/usr/local/bin/va-restore-full YYYYMMDD_HHMMSS

# 5. Verify data integrity
# Run data validation scripts

# 6. Resume operations
docker compose start voiceassist-server

Scenario 3: Accidental Data Deletion

# Restore specific component only (faster than full restore)

# For deleted PostgreSQL table/data:
BACKUP_FILE="/backups/postgres/daily/voiceassist_20251121_120000.dump.gz"
# Use table-specific restore procedure

# For deleted Redis data:
# Use Redis restore procedure

# For deleted Qdrant collection:
# Use Qdrant restore procedure

Backup Monitoring

Backup Health Check

#!/bin/bash
# Save as: /usr/local/bin/va-backup-health

BACKUP_ROOT="/backups"
ALERT_EMAIL="ops-team@voiceassist.local"

echo "Backup Health Check - $(date)"
echo "========================================"

# Check PostgreSQL backups
LATEST_PG=$(find ${BACKUP_ROOT}/postgres/daily -name "*.dump.gz" -mtime -1 | wc -l)
if [ $LATEST_PG -eq 0 ]; then
    echo "⚠️  WARNING: No PostgreSQL backup in last 24 hours"
else
    echo "✓ PostgreSQL backups are current"
fi

# Check Redis backups
LATEST_REDIS=$(find ${BACKUP_ROOT}/redis -name "*.rdb" -mtime -1 | wc -l)
if [ $LATEST_REDIS -eq 0 ]; then
    echo "⚠️  WARNING: No Redis backup in last 24 hours"
else
    echo "✓ Redis backups are current"
fi

# Check Qdrant backups
LATEST_QDRANT=$(find ${BACKUP_ROOT}/qdrant -name "*.snapshot" -mtime -1 | wc -l)
if [ $LATEST_QDRANT -eq 0 ]; then
    echo "⚠️  WARNING: No Qdrant backup in last 24 hours"
else
    echo "✓ Qdrant backups are current"
fi

# Check disk space
DISK_USAGE=$(df -h ${BACKUP_ROOT} | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 80 ]; then
    echo "⚠️  WARNING: Backup disk usage at ${DISK_USAGE}%"
else
    echo "✓ Backup disk space is adequate (${DISK_USAGE}%)"
fi

# Check backup sizes
echo ""
echo "Backup Sizes:"
echo "PostgreSQL: $(du -sh ${BACKUP_ROOT}/postgres | cut -f1)"
echo "Redis: $(du -sh ${BACKUP_ROOT}/redis | cut -f1)"
echo "Qdrant: $(du -sh ${BACKUP_ROOT}/qdrant | cut -f1)"
echo "Config: $(du -sh ${BACKUP_ROOT}/config | cut -f1)"
echo "Total: $(du -sh ${BACKUP_ROOT} | cut -f1)"

Related Documentation

• Deployment Runbook
• Incident Response Runbook
• Troubleshooting Runbook
• Monitoring Runbook
• UNIFIED_ARCHITECTURE.md

Document Version: 1.0 Last Updated: 2025-11-21 Maintained By: VoiceAssist DevOps Team Review Cycle: Quarterly or after each disaster recovery event Next Review: 2026-02-21

Scaling Runbook

Last Updated: 2025-11-27 Purpose: Comprehensive guide for scaling VoiceAssist V2 infrastructure

Scaling Overview

Current Architecture

Load Balancer (if configured)
    ↓
VoiceAssist Server (Scalable)
    ↓
├── PostgreSQL (Primary + Read Replicas)
├── Redis (Cluster or Sentinel)
└── Qdrant (Distributed)

Scaling Strategy

When to Scale

Scaling Triggers

Immediate Scaling (Reactive)

Scale immediately if:

• CPU usage > 80% for 10+ minutes
• Memory usage > 85%
• Response time > 2 seconds (p95)
• Error rate > 5%
• Connection pool exhausted
• Queue depth > 1000

Planned Scaling (Proactive)

Schedule scaling if:

• Expected traffic increase (events, marketing campaigns)
• New feature launch with heavy load
• Approaching 70% capacity on any metric
• Seasonal traffic patterns

Scaling Decision Matrix

# Quick capacity check
cat > /usr/local/bin/va-capacity-check <<'EOF'
#!/bin/bash

echo "VoiceAssist Capacity Check - $(date)"
echo "========================================"

# Check application load
CPU=$(docker stats --no-stream --format "{{.CPUPerc}}" voiceassist-voiceassist-server-1 | sed 's/%//')
MEM=$(docker stats --no-stream --format "{{.MemPerc}}" voiceassist-voiceassist-server-1 | sed 's/%//')

echo "Application:"
echo "  CPU: ${CPU}%"
echo "  Memory: ${MEM}%"

# Database connections
DB_CONN=$(docker compose exec -T postgres psql -U voiceassist -d voiceassist -t -c \
  "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';" | tr -d ' ')
DB_MAX=$(docker compose exec -T postgres psql -U voiceassist -d voiceassist -t -c \
  "SHOW max_connections;" | tr -d ' ')
DB_USAGE=$((DB_CONN * 100 / DB_MAX))

echo "Database:"
echo "  Active Connections: ${DB_CONN}/${DB_MAX} (${DB_USAGE}%)"

# Redis memory
REDIS_MEM=$(docker compose exec -T redis redis-cli INFO memory | grep used_memory_human | cut -d: -f2 | tr -d '\r')
echo "Redis:"
echo "  Memory Usage: ${REDIS_MEM}"

# Recommendation
echo ""
echo "Scaling Recommendations:"
if (( $(echo "$CPU > 80" | bc -l) )) || (( $(echo "$MEM > 85" | bc -l) )); then
    echo "🔴 IMMEDIATE: Scale application horizontally"
elif (( $(echo "$CPU > 70" | bc -l) )) || (( $(echo "$MEM > 70" | bc -l) )); then
    echo "🟡 SOON: Plan to scale within 24 hours"
elif [ $DB_USAGE -gt 80 ]; then
    echo "🔴 IMMEDIATE: Scale database connections or add read replica"
else
    echo "🟢 OK: Current capacity is adequate"
fi
EOF

chmod +x /usr/local/bin/va-capacity-check

Horizontal Scaling - Application Server

Quick Scale Up

# Scale to 3 instances
docker compose up -d --scale voiceassist-server=3

# Verify all instances running
docker compose ps voiceassist-server

# Expected output: 3 containers running
# voiceassist-voiceassist-server-1
# voiceassist-voiceassist-server-2
# voiceassist-voiceassist-server-3

# Check health of all instances
for i in {1..3}; do
    echo "Instance $i:"
    docker inspect voiceassist-voiceassist-server-$i | jq '.[0].State.Health.Status'
done

Scale with Load Balancer

# Add to docker-compose.yml
services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - voiceassist-server

  voiceassist-server:
    # ... existing config ...
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "2"
          memory: 2G
        reservations:
          cpus: "1"
          memory: 1G

# Create nginx.conf for load balancing
upstream voiceassist_backend {
    least_conn;  # Use least connections algorithm

    server voiceassist-server-1:8000 max_fails=3 fail_timeout=30s;
    server voiceassist-server-2:8000 max_fails=3 fail_timeout=30s;
    server voiceassist-server-3:8000 max_fails=3 fail_timeout=30s;

    keepalive 32;
}

server {
    listen 80;

    location / {
        proxy_pass http://voiceassist_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # Timeouts
        proxy_connect_timeout 5s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        # Health check
        proxy_next_upstream error timeout http_500 http_502 http_503;
    }

    location /health {
        access_log off;
        proxy_pass http://voiceassist_backend;
    }
}