2:I[7012,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],"MarkdownRenderer"]
4:I[9856,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],""]
5:I[4126,[],""]
7:I[9630,[],""]
8:I[4278,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"HeadingProvider"]
9:I[1476,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Header"]
a:I[3167,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Sidebar"]
b:I[7409,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"PageFrame"]
3:T29d4,
# Epic FHIR Integration Operations Runbook

## Overview

This runbook covers operational procedures for the Epic FHIR integration (Phase 6 & 6b) in VoiceAssist Voice Mode.

## Quick Reference

| Component        | Location                                   | Health Check          |
| ---------------- | ------------------------------------------ | --------------------- |
| FHIR Client      | `integrations/fhir/fhir_client.py`         | `/health/epic`        |
| Epic Adapter     | `integrations/fhir/epic_adapter.py`        | Check token status    |
| Provider Monitor | `integrations/fhir/provider_monitor.py`    | Circuit breaker state |
| EHR Commands     | `dictation_engine/plugins/ehr_commands.py` | Plugin status         |

---

## 1. Epic Credential Rotation

### When to Rotate

- Every 90 days (recommended)
- After security incidents
- When staff with access leave organization

### Rotation Procedure

1. **Generate New Credentials in Epic**

   ```
   - Log into Epic App Orchard
   - Navigate to your application
   - Generate new client credentials
   - Download new private key
   ```

2. **Update Environment Variables**

   ```bash
   # Back up current credentials
   cp /opt/voiceassist/.env /opt/voiceassist/.env.backup.$(date +%Y%m%d)

   # Update credentials
   export EPIC_CLIENT_ID="new_client_id"
   export EPIC_PRIVATE_KEY_PATH="/path/to/new/private_key.pem"
   ```

3. **Verify New Credentials**

   ```bash
   # Test token generation
   curl -X POST "$EPIC_TOKEN_URL" \
     -H "Content-Type: application/x-www-form-urlencoded" \
     -d "grant_type=client_credentials&client_id=$EPIC_CLIENT_ID"
   ```

4. **Restart Services**

   ```bash
   sudo systemctl restart voiceassist-api
   ```

5. **Verify Connectivity**

   ```bash
   curl http://localhost:5057/health/epic
   ```

6. **Revoke Old Credentials**
   - Return to Epic App Orchard
   - Revoke the previous credentials

---

## 2. Provider Outage Recovery

### Symptoms

- Circuit breaker in OPEN state
- High error rate (>10%)
- Provider status: UNHEALTHY

### Investigation

1. **Check Provider Status**

   ```bash
   curl http://localhost:5057/api/admin/provider-status
   ```

2. **Check Circuit Breaker**

   ```python
   # In Python console
   from app.integrations.fhir import get_epic_adapter
   adapter = get_epic_adapter()
   print(adapter.provider_monitor.get_status())
   ```

3. **Check Epic Status Page**
   - Visit Epic's status page for known outages
   - Check App Orchard for maintenance windows

### Recovery Steps

1. **If Epic is Down**
   - Fallback mode activates automatically
   - Monitor `degradation.activated` events
   - Use cached patient context

2. **If Circuit Breaker is Open**

   ```python
   # Wait for half-open transition (60 seconds default)
   # Or manually reset if Epic is confirmed healthy
   monitor = adapter.provider_monitor
   monitor._circuit_breaker._transition_to(CircuitState.HALF_OPEN)
   ```

3. **Force Health Check**

   ```python
   result = await adapter.provider_monitor.check_health()
   print(result)
   ```

4. **Verify Recovery**

   ```bash
   # Check provider is healthy
   curl http://localhost:5057/health/epic

   # Check metrics
   curl http://localhost:5057/api/admin/metrics/epic
   ```

---

## 3. Feature Flag Management

### Enable/Disable Epic Write Operations

1. **Via Admin API**

   ```bash
   # Disable write operations
   curl -X POST http://localhost:5057/api/admin/features \
     -H "Content-Type: application/json" \
     -d '{"feature": "epic_fhir_write", "enabled": false}'
   ```

2. **Via Code**

   ```python
   from app.core.policy_config import get_policy_service
   policy = get_policy_service()
   policy.update_feature_flag("epic_fhir_write", False)
   ```

3. **Per-User Override**
   ```python
   policy.set_user_override("user_123", "epic_fhir_write", True)
   ```

### A/B Test Monitoring

```bash
# Get A/B test variants for a user
curl "http://localhost:5057/api/admin/ab-variant?user_id=user_123&test=epic_fhir_write"
```

---

## 4. Handling Write Operation Failures

### Order Submission Failures

1. **Check Audit Logs**

   ```python
   from app.core.audit_service import get_audit_service
   audit = get_audit_service()
   events = await audit.get_events(
       event_type=AuditEventType.EHR_WRITE_FAILED,
       limit=10
   )
   ```

2. **Common Error Codes**
   | Code | Meaning | Action |
   |------|---------|--------|
   | 400 | Validation error | Check resource format |
   | 401 | Auth failed | Rotate credentials |
   | 403 | Authorization denied | Check scopes |
   | 409 | Conflict | Duplicate order detected |
   | 412 | Precondition failed | ETag mismatch, retry |
   | 429 | Rate limited | Wait and retry |
   | 500+ | Server error | Check Epic status |

3. **Manual Order Recovery**
   - Log into Epic directly
   - Verify order status
   - Create order manually if needed
   - Document in incident report

---

## 5. Data Export Requests

### User Data Export (HIPAA Compliance)

1. **Generate Export**

   ```python
   from app.core.audit_service import get_audit_service
   audit = get_audit_service()

   # Get user's PHI access log
   events = await audit.get_events(user_id="user_123", limit=10000)

   # Export to JSON
   export = await audit.export_audit_log(
       start_time=datetime(2024, 1, 1),
       format="json"
   )
   ```

2. **Accounting of Disclosures**
   ```python
   accounting = await audit.get_accounting_of_disclosures(
       patient_id="patient_456"
   )
   ```

---

## 6. Chaos Engineering Tests

### Running Predefined Experiments

```python
from app.testing import get_chaos_controller, create_epic_outage_experiment

chaos = get_chaos_controller()
chaos.enable()  # Only in test environment!

# Simulate Epic outage
experiment = create_epic_outage_experiment(duration_seconds=60)
chaos._experiments[experiment.id] = experiment
results = await chaos.run_experiment(experiment.id)

chaos.disable()
```

### Manual Chaos Injection

```python
# Inject latency
chaos.inject_latency("epic", min_ms=500, max_ms=2000, duration_seconds=300)

# Inject errors
chaos.inject_error_rate("epic", rate=0.3, duration_seconds=300)

# Clear all chaos
chaos.clear_all()
```

---

## 7. Monitoring Dashboards

### Key Metrics to Monitor

| Metric             | Healthy | Warning    | Critical |
| ------------------ | ------- | ---------- | -------- |
| Epic Success Rate  | >99%    | 95-99%     | <95%     |
| Avg Latency        | <500ms  | 500-2000ms | >2000ms  |
| Write Success Rate | >99%    | 95-99%     | <95%     |
| Circuit State      | CLOSED  | HALF_OPEN  | OPEN     |

### Log Analysis

```bash
# Epic errors in last hour
grep "epic" /var/log/voiceassist/api.log | grep -i error | tail -100

# Write operation failures
grep "EHR_WRITE_FAILED" /var/log/voiceassist/audit.log | tail -50
```

---

## 8. Emergency Procedures

### Complete Epic Shutdown

1. **Disable All Epic Features**

   ```bash
   curl -X POST http://localhost:5057/api/admin/features/batch \
     -H "Content-Type: application/json" \
     -d '{
       "updates": [
         {"feature": "epic_fhir_read_only", "enabled": false},
         {"feature": "epic_fhir_write", "enabled": false}
       ]
     }'
   ```

2. **Verify Fallback Active**

   ```bash
   curl http://localhost:5057/health/epic
   # Should show fallback_active: true
   ```

3. **Notify Users**
   - System automatically uses cached context
   - Voice queries return "EHR temporarily unavailable"

### Recovery from Emergency Shutdown

1. Verify Epic connectivity restored
2. Re-enable read operations first
3. Monitor for 15 minutes
4. Re-enable write operations
5. Monitor for 30 minutes
6. Document incident

---

## 9. GDPR/CCPA Compliance Workflows

### Data Subject Access Request (DSAR)

1. **Identify User Data**

   ```python
   from app.core.audit_service import get_audit_service

   audit = get_audit_service()

   # Get all audit events for a user
   user_events = await audit.get_events(
       user_id="user_123",
       limit=100000
   )

   # Get EHR access history
   ehr_events = await audit.get_events(
       user_id="user_123",
       event_type_prefix="ehr."
   )
   ```

2. **Generate Export**
   ```python
   export = await audit.export_user_data(
       user_id="user_123",
       format="json",
       include_ehr_access=True
   )
   ```

### Data Deletion Request (Right to Erasure)

1. **Verify Request**
   - Confirm identity of requestor
   - Check for legal retention requirements
   - Document the request in audit log

2. **Delete User Data**

   ```python
   from app.core.audit_service import get_audit_service

   audit = get_audit_service()

   # Log the deletion request
   await audit.log_event(
       event_type=AuditEventType.DATA_DELETION_REQUESTED,
       user_id="user_123",
       details={"reason": "GDPR erasure request", "ticket": "GDPR-2025-001"}
   )

   # Delete EHR-related data (preserve audit logs per HIPAA)
   # Note: Audit logs must be retained for 6 years per HIPAA
   ```

3. **Confirmation**
   - Send confirmation to user
   - Update data retention records

### Audit Log Retention Policy

| Data Type            | Retention Period | Legal Basis             |
| -------------------- | ---------------- | ----------------------- |
| EHR Access Logs      | 6 years          | HIPAA §164.530(j)       |
| Write Operation Logs | 6 years          | HIPAA §164.530(j)       |
| User Session Data    | 1 year           | Business requirement    |
| Error Logs           | 90 days          | Operational requirement |
| Chaos Test Results   | 30 days          | Testing documentation   |

### Accounting of Disclosures

```python
# Generate accounting of disclosures report (HIPAA requirement)
accounting = await audit.get_accounting_of_disclosures(
    patient_id="patient_456",
    start_date=datetime(2024, 1, 1),
    end_date=datetime.now()
)

# Export for patient request
report = await audit.export_accounting_of_disclosures(
    patient_id="patient_456",
    format="pdf"
)
```

---

## 10. Contact Information

| Role             | Contact                  |
| ---------------- | ------------------------ |
| Epic Support     | epic-support@example.com |
| On-Call Engineer | oncall@voiceassist.io    |
| Security Team    | security@voiceassist.io  |

---

## Appendix: Common Commands

```bash
# Check all service health
curl http://localhost:5057/health

# Get Epic adapter status
curl http://localhost:5057/api/admin/epic/status

# List active feature flags
curl http://localhost:5057/api/admin/features

# Get audit statistics
curl http://localhost:5057/api/admin/audit/stats

# Check chaos controller status
curl http://localhost:5057/api/admin/chaos/status
```
6:["slug","operations/epic-fhir-runbook","c"]
0:["X7oMT3VrOffzp0qvbeOas",[[["",{"children":["docs",{"children":[["slug","operations/epic-fhir-runbook","c"],{"children":["__PAGE__?{\"slug\":[\"operations\",\"epic-fhir-runbook\"]}",{}]}]}]},"$undefined","$undefined",true],["",{"children":["docs",{"children":[["slug","operations/epic-fhir-runbook","c"],{"children":["__PAGE__",{},[["$L1",["$","div",null,{"children":[["$","div",null,{"className":"mb-6 flex items-center justify-between gap-4","children":[["$","div",null,{"children":[["$","p",null,{"className":"text-sm text-gray-500 dark:text-gray-400","children":"Docs / Raw"}],["$","h1",null,{"className":"text-3xl font-bold text-gray-900 dark:text-white","children":"Epic FHIR Integration Operations Runbook"}],["$","p",null,{"className":"text-sm text-gray-600 dark:text-gray-400","children":["Sourced from"," ",["$","code",null,{"className":"font-mono text-xs","children":["docs/","operations/epic-fhir-runbook.md"]}]]}]]}],["$","a",null,{"href":"https://github.com/mohammednazmy/VoiceAssist/edit/main/docs/operations/epic-fhir-runbook.md","target":"_blank","rel":"noreferrer","className":"inline-flex items-center gap-2 rounded-md border border-gray-200 dark:border-gray-700 px-3 py-1.5 text-sm text-gray-700 dark:text-gray-200 hover:border-primary-500 dark:hover:border-primary-400 hover:text-primary-700 dark:hover:text-primary-300","children":"Edit on GitHub"}]]}],["$","div",null,{"className":"rounded-lg border border-gray-200 dark:border-gray-800 bg-white dark:bg-gray-900 p-6","children":["$","$L2",null,{"content":"$3"}]}],["$","div",null,{"className":"mt-6 flex flex-wrap gap-2 text-sm","children":[["$","$L4",null,{"href":"/reference/all-docs","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"← All documentation"}],["$","$L4",null,{"href":"/","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"Home"}]]}]]}],null],null],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children","$6","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/_next/static/css/7f586cdbbaa33ff7.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","className":"h-full","children":["$","body",null,{"className":"__className_f367f3 h-full bg-white dark:bg-gray-900","children":[["$","a",null,{"href":"#main-content","className":"skip-to-content","children":"Skip to main content"}],["$","$L8",null,{"children":[["$","$L9",null,{}],["$","$La",null,{}],["$","main",null,{"id":"main-content","className":"lg:pl-64","role":"main","aria-label":"Documentation content","children":["$","$Lb",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]]}]]}]}]],null],null],["$Lc",null]]]]
c:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"Epic FHIR Integration Operations Runbook | Docs | VoiceAssist Docs"}],["$","meta","3",{"name":"description","content":"Operational procedures for Epic FHIR integration including credential rotation, provider outage recovery, feature flag management, write operation failures, chaos engineering, and emergency procedures."}],["$","meta","4",{"name":"keywords","content":"VoiceAssist,documentation,medical AI,voice assistant,healthcare,HIPAA,API"}],["$","meta","5",{"name":"robots","content":"index, follow"}],["$","meta","6",{"name":"googlebot","content":"index, follow"}],["$","link","7",{"rel":"canonical","href":"https://assistdocs.asimo.io"}],["$","meta","8",{"property":"og:title","content":"VoiceAssist Documentation"}],["$","meta","9",{"property":"og:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","10",{"property":"og:url","content":"https://assistdocs.asimo.io"}],["$","meta","11",{"property":"og:site_name","content":"VoiceAssist Docs"}],["$","meta","12",{"property":"og:type","content":"website"}],["$","meta","13",{"name":"twitter:card","content":"summary"}],["$","meta","14",{"name":"twitter:title","content":"VoiceAssist Documentation"}],["$","meta","15",{"name":"twitter:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","16",{"name":"next-size-adjust"}]]
1:null