2:I[7012,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],"MarkdownRenderer"] 4:I[9856,["4765","static/chunks/4765-f5afdf8061f456f3.js","9856","static/chunks/9856-3b185291364d9bef.js","6687","static/chunks/app/docs/%5B...slug%5D/page-e07536548216bee4.js"],""] 5:I[4126,[],""] 7:I[9630,[],""] 8:I[4278,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"HeadingProvider"] 9:I[1476,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Header"] a:I[3167,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"Sidebar"] b:I[7409,["9856","static/chunks/9856-3b185291364d9bef.js","8172","static/chunks/8172-b3a2d6fe4ae10d40.js","3185","static/chunks/app/layout-2814fa5d15b84fe4.js"],"PageFrame"] 3:T3189, # VoiceAssist Project Summary ## Executive Overview VoiceAssist is a comprehensive AI assistant system designed specifically for medical professionals. It combines voice and text interfaces with specialized medical knowledge retrieval, system integrations, and privacy-conscious architecture. ### Key Differentiators 1. **Medical Specialization**: Access to medical textbooks, journals, clinical guidelines 2. **Hybrid AI Architecture**: Local models for privacy + cloud APIs for power 3. **Always-On Voice**: Natural conversation with wake word activation 4. **System Integration**: Calendar, email, files, reminders, notes 5. **Privacy-First**: HIPAA-conscious design, PHI stays local 6. **Multi-Platform**: macOS native, web, iOS (planned) ## System Architecture at a Glance ``` ┌──────────────────────────────────────────────────────────┐ │ User Interfaces │ │ macOS Client | Web App | iOS App (future) | Admin Panel │ └────────────────────────┬─────────────────────────────────┘ │ HTTPS/WSS │ ┌────────────────────────┴─────────────────────────────────┐ │ Ubuntu Server (asimo.io) │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌────────────────┐ │ │ │ API Server │ │ Medical KB │ │ Integrations │ │ │ │ (FastAPI) │ │ (RAG/Vector)│ │ (Nextcloud │ │ │ │ │ │ │ │ PubMed, etc) │ │ │ └──────────────┘ └──────────────┘ └────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────┐│ │ │ Data Layer: PostgreSQL + Redis + Qdrant ││ │ └──────────────────────────────────────────────────────┘│ └────────────────────────────────────────────────────────────┘ ``` ## Technology Stack Summary ### Backend - **Language**: Python 3.11+ - **Framework**: FastAPI - **Databases**: PostgreSQL (pgvector), Redis, Qdrant - **Task Queue**: Celery - **Server**: Nginx, Docker, Ubuntu 20.04+ ### Frontend - **Framework**: React 18+ with TypeScript - **Build Tool**: Vite - **Styling**: Tailwind CSS - **Components**: shadcn/ui, Tremor (admin) - **State**: Zustand - **Communication**: WebSocket, REST ### AI/ML - **Cloud**: OpenAI GPT-4, OpenAI Realtime API - **Local**: Ollama (Llama 3.1) - **Embeddings**: OpenAI text-embedding-3-large - **Voice**: OpenAI Realtime API, Porcupine (wake word) ### Infrastructure - **Hosting**: Ubuntu server at asimo.io - **Reverse Proxy**: Nginx - **SSL**: Let's Encrypt - **Containerization**: Docker - **Monitoring**: Prometheus + Grafana (planned) ## Component Breakdown ### 1. Ubuntu Server (asimo.io) **Purpose**: Central backend services, medical knowledge base, integrations **Services**: - Voice API (port 8001) - WebSocket proxy to OpenAI Realtime - Medical KB API (port 8002) - RAG and literature search - Main API (port 8000) - Chat, auth, general endpoints - Admin API (port 8003) - System management **Subdomains**: - `voice.asimo.io` - Voice service - `medical-kb.asimo.io` - Medical knowledge - `voiceassist.asimo.io` - Main web app - `admin.asimo.io` - Admin panel - `docs-voice.asimo.io` - Documentation ### 2. macOS Native Client **Purpose**: Always-on voice assistant with system integration **Features**: - Wake word detection (Porcupine) - Menu bar application - Local LLM (Ollama) - File system indexing - macOS system integrations (Calendar, Mail, Files, etc.) - Privacy classification **Tech**: Python + Swift/SwiftUI (or Electron) ### 3. Web Application **Purpose**: Browser-based access from any device **Features**: - Text and voice chat - File uploads - Citation display - Conversation history - Responsive design **Tech**: React + TypeScript + Vite ### 4. Admin Panel **Purpose**: System management and monitoring **Features**: - Real-time dashboard - AI model configuration - Knowledge base management - Integration control - Analytics and logs - Backup management **Tech**: React + TypeScript + Tremor ### 5. Documentation Site **Purpose**: User and admin documentation **Features**: - User guides - Medical features documentation - API reference - Troubleshooting - Search **Tech**: Next.js + MDX ## Medical Features ### Medical Textbook Knowledge Base - Upload PDFs of medical textbooks - Automatic indexing with RAG - Semantic search with citations - Page-level references - Voice narration **Example Sources**: - Harrison's Internal Medicine - Robbins Pathology - UpToDate (if subscription) - Specialty textbooks ### Medical Journal Search - PubMed API integration - Automatic PDF download (PMC) - Full-text indexing - Evidence synthesis - Citation formatting ### Clinical Guidelines - CDC, WHO, specialty societies - Automatic updates - Version tracking - Recommendation strength ### OpenEvidence Integration - Clinical question answering - Evidence summaries - Cross-reference with literature ### Medical Calculators - CHA2DS2-VASc, HAS-BLED, etc. - Voice-activated - Integrated with guidelines ## Privacy & Security Architecture ### Data Classification Tiers **Tier 1 - Strictly Local (PHI)** - Patient notes, medical records - Sensitive personal files - Never sent to external APIs - Processed by local Ollama only **Tier 2 - Server (Private)** - Personal documents - Email/calendar - Stored on Ubuntu server - Not sent to commercial APIs **Tier 3 - Cloud OK (Public)** - Medical literature queries - General medical knowledge - Web searches - Can use OpenAI/Claude ### Privacy Mechanisms - Automatic PHI detection (keywords, patterns) - File path classification - User-marked sensitivity - Default to strictest tier - Audit logging ### Security Measures - HTTPS/WSS only - JWT authentication - API rate limiting - Encrypted storage - Regular backups to Nextcloud - Firewall (UFW) - SSL certificates (Let's Encrypt) ## Development Roadmap ### Phase 1: Foundation (Weeks 1-2) - Ubuntu server setup - Docker, PostgreSQL, Redis, Qdrant - Subdomain configuration - SSL certificates - Basic FastAPI structure ### Phase 2: Core Voice Interface (Weeks 3-4) - macOS client setup - Wake word detection - OpenAI Realtime API integration - Basic orchestrator - LaunchAgent auto-start ### Phase 3: Local AI & System Integration (Weeks 5-6) - Ollama setup and integration - macOS system integrations (Calendar, Mail, Files, etc.) - Local file indexing - Privacy classifier ### Phase 4: Medical Knowledge Base (Weeks 7-9) - Vector database (Qdrant) - PDF processing pipeline - RAG system - PubMed API integration - Textbook indexing - Guidelines scraping ### Phase 5: Web Application (Weeks 10-12) - React app setup - Chat interface with streaming - Voice integration (Web Audio) - File upload - Responsive design ### Phase 6: Admin Panel (Weeks 13-14) - Dashboard with real-time metrics - Model configuration UI - Knowledge base management - Integration controls - Analytics and logs ### Phase 7: Nextcloud Integration (Week 15) - WebDAV integration - Auto-indexing - Backup integration ### Phase 8: Documentation Site (Week 16) - Next.js site setup - Content creation - Search integration - Deployment ### Phase 9: Testing & Refinement (Weeks 17-18) - Integration testing - Performance optimization - Security audit - Bug fixes ### Phase 10: iOS Support (Weeks 19-21) - Optional - iOS app development - Or Shortcuts integration ## Key Files Reference ### Documentation - [README.md](../README.md) - Project overview - [ARCHITECTURE.md](./ARCHITECTURE.md) - Detailed architecture - [ROADMAP.md](./ROADMAP.md) - Development plan - [MEDICAL_FEATURES.md](./MEDICAL_FEATURES.md) - Medical capabilities - [INFRASTRUCTURE_SETUP.md](./INFRASTRUCTURE_SETUP.md) - Server setup guide - [WEB_APP_SPECS.md](./WEB_APP_SPECS.md) - Web app specifications - [ADMIN_PANEL_SPECS.md](./ADMIN_PANEL_SPECS.md) - Admin panel specs - [DOCUMENTATION_SITE_SPECS.md](./DOCUMENTATION_SITE_SPECS.md) - Docs site specs ### Component READMEs - [server/README.md](../server/README.md) - Backend server - [macos-client/README.md](../macos-client/README.md) - macOS client - [apps/web-app/README.md](../apps/web-app/README.md) - Web application - [apps/admin-panel/README.md](../apps/admin-panel/README.md) - Admin panel - [apps/docs-site/README.md](../apps/docs-site/README.md) - Documentation site ## Quick Start Commands ### Server Setup (Ubuntu) ```bash # Install Docker curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh # Install PostgreSQL sudo apt install postgresql postgresql-contrib # Install Redis sudo apt install redis-server # Clone and setup cd ~/VoiceAssist/server python3 -m venv venv source venv/bin/activate pip install -r requirements.txt # Start services uvicorn app.main:app --reload ``` ### macOS Client Setup ```bash # Install Ollama brew install ollama # Download model ollama pull llama3.1:8b # Setup client cd ~/VoiceAssist/macos-client pip3 install -r requirements.txt # Run python3 main.py ``` ### Web App Development ```bash cd ~/VoiceAssist/apps/web-app npm install npm run dev ``` ### Admin Panel Development ```bash cd ~/VoiceAssist/apps/admin-panel npm install npm run dev ``` ### Documentation Site ```bash cd ~/VoiceAssist/apps/docs-site npm install npm run dev ``` ## Environment Variables Quick Reference ### Server (.env) ```bash DATABASE_URL=postgresql://user:pass@localhost/voiceassist REDIS_URL=redis://localhost:6379 QDRANT_URL=http://localhost:6333 OPENAI_API_KEY=sk-... SECRET_KEY=... JWT_SECRET=... ``` ### macOS Client (.env) ```bash SERVER_URL=https://voice.asimo.io PORCUPINE_ACCESS_KEY=... OPENAI_API_KEY=sk-... OLLAMA_URL=http://localhost:11434 WAKE_WORD=computer ``` ### Web Apps (.env.local) ```bash VITE_API_URL=https://voice.asimo.io VITE_WS_URL=wss://voice.asimo.io/ws VITE_ENV=development ``` ## Success Metrics ### Performance Targets - Voice activation latency: < 500ms - Response time (simple): < 2s - Medical search (with RAG): < 5s - Uptime: > 99.5% - Voice recognition accuracy: > 95% ### Quality Targets - Citation accuracy: 100% - PHI leakage: 0 incidents - User satisfaction: High ## Cost Estimates (Monthly) ### Cloud Services - OpenAI API (GPT-4): ~$40-60 - OpenAI Realtime API: ~$10-20 - OpenAI Embeddings: ~$5-10 - OpenEvidence: ~$10 (subscription) ### Infrastructure - Server: $0 (owned) - Domain: $15/year - SSL: $0 (Let's Encrypt) **Total Monthly**: ~$60-100 ### Cost Optimization - Use local model for routine queries - Batch embedding generation - Cache common responses - Implement request deduplication ## Next Steps 1. **Review all documentation** to ensure understanding 2. **Set up Ubuntu server** following INFRASTRUCTURE_SETUP.md 3. **Configure DNS** for subdomains 4. **Install prerequisites** on macOS 5. **Start with Phase 1** of roadmap 6. **Iterate and test** each component 7. **Deploy incrementally** to production ## Support & Resources ### Documentation - Full docs: https://docs-voice.asimo.io (when deployed) - GitHub repo: (to be created) ### APIs Used - OpenAI: https://platform.openai.com/docs - PubMed E-utilities: https://www.ncbi.nlm.nih.gov/books/NBK25501/ - Ollama: https://ollama.ai/docs - Qdrant: https://qdrant.tech/documentation/ ### Communities - OpenAI Discord - FastAPI Discord - React community ## License Personal use project. All medical content sources must respect their individual licenses. --- **Project Started**: November 19, 2024 **Primary Developer**: Dr. Mohammed Nazmy **Status**: Planning Phase **Version**: 0.1.0 (planning) 6:["slug","PROJECT_SUMMARY","c"] 0:["X7oMT3VrOffzp0qvbeOas",[[["",{"children":["docs",{"children":[["slug","PROJECT_SUMMARY","c"],{"children":["__PAGE__?{\"slug\":[\"PROJECT_SUMMARY\"]}",{}]}]}]},"$undefined","$undefined",true],["",{"children":["docs",{"children":[["slug","PROJECT_SUMMARY","c"],{"children":["__PAGE__",{},[["$L1",["$","div",null,{"children":[["$","div",null,{"className":"mb-6 flex items-center justify-between gap-4","children":[["$","div",null,{"children":[["$","p",null,{"className":"text-sm text-gray-500 dark:text-gray-400","children":"Docs / Raw"}],["$","h1",null,{"className":"text-3xl font-bold text-gray-900 dark:text-white","children":"Project Summary"}],["$","p",null,{"className":"text-sm text-gray-600 dark:text-gray-400","children":["Sourced from"," ",["$","code",null,{"className":"font-mono text-xs","children":["docs/","PROJECT_SUMMARY.md"]}]]}]]}],["$","a",null,{"href":"https://github.com/mohammednazmy/VoiceAssist/edit/main/docs/PROJECT_SUMMARY.md","target":"_blank","rel":"noreferrer","className":"inline-flex items-center gap-2 rounded-md border border-gray-200 dark:border-gray-700 px-3 py-1.5 text-sm text-gray-700 dark:text-gray-200 hover:border-primary-500 dark:hover:border-primary-400 hover:text-primary-700 dark:hover:text-primary-300","children":"Edit on GitHub"}]]}],["$","div",null,{"className":"rounded-lg border border-gray-200 dark:border-gray-800 bg-white dark:bg-gray-900 p-6","children":["$","$L2",null,{"content":"$3"}]}],["$","div",null,{"className":"mt-6 flex flex-wrap gap-2 text-sm","children":[["$","$L4",null,{"href":"/reference/all-docs","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"← All documentation"}],["$","$L4",null,{"href":"/","className":"inline-flex items-center gap-1 rounded-md bg-gray-100 px-3 py-1 text-gray-700 hover:bg-gray-200 dark:bg-gray-800 dark:text-gray-200 dark:hover:bg-gray-700","children":"Home"}]]}]]}],null],null],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children","$6","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[null,["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children","docs","children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":"$undefined","notFoundStyles":"$undefined"}]],null]},[[[["$","link","0",{"rel":"stylesheet","href":"/_next/static/css/7f586cdbbaa33ff7.css","precedence":"next","crossOrigin":"$undefined"}]],["$","html",null,{"lang":"en","className":"h-full","children":["$","body",null,{"className":"__className_f367f3 h-full bg-white dark:bg-gray-900","children":[["$","a",null,{"href":"#main-content","className":"skip-to-content","children":"Skip to main content"}],["$","$L8",null,{"children":[["$","$L9",null,{}],["$","$La",null,{}],["$","main",null,{"id":"main-content","className":"lg:pl-64","role":"main","aria-label":"Documentation content","children":["$","$Lb",null,{"children":["$","$L5",null,{"parallelRouterKey":"children","segmentPath":["children"],"error":"$undefined","errorStyles":"$undefined","errorScripts":"$undefined","template":["$","$L7",null,{}],"templateStyles":"$undefined","templateScripts":"$undefined","notFound":[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":"404"}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],"notFoundStyles":[]}]}]}]]}]]}]}]],null],null],["$Lc",null]]]] c:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","title","2",{"children":"Project Summary | Docs | VoiceAssist Docs"}],["$","meta","3",{"name":"description","content":"VoiceAssist is a comprehensive AI assistant system designed specifically for medical professionals. It combines voice and text interfaces with special..."}],["$","meta","4",{"name":"keywords","content":"VoiceAssist,documentation,medical AI,voice assistant,healthcare,HIPAA,API"}],["$","meta","5",{"name":"robots","content":"index, follow"}],["$","meta","6",{"name":"googlebot","content":"index, follow"}],["$","link","7",{"rel":"canonical","href":"https://assistdocs.asimo.io"}],["$","meta","8",{"property":"og:title","content":"VoiceAssist Documentation"}],["$","meta","9",{"property":"og:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","10",{"property":"og:url","content":"https://assistdocs.asimo.io"}],["$","meta","11",{"property":"og:site_name","content":"VoiceAssist Docs"}],["$","meta","12",{"property":"og:type","content":"website"}],["$","meta","13",{"name":"twitter:card","content":"summary"}],["$","meta","14",{"name":"twitter:title","content":"VoiceAssist Documentation"}],["$","meta","15",{"name":"twitter:description","content":"Comprehensive documentation for VoiceAssist - Enterprise Medical AI Assistant"}],["$","meta","16",{"name":"next-size-adjust"}]] 1:null