Nextcloud Integration Guide

Overview

VoiceAssist integrates with Nextcloud for identity management, file storage, calendar, and email functionality.

Current Status (Phase 6): VoiceAssist now has working CalDAV calendar integration, WebDAV file auto-indexing, and email service skeleton.

Implementation Notes:

Phase 2: Nextcloud added to docker-compose.yml, OCS API integration created
Phase 6: CalDAV calendar operations, WebDAV file auto-indexer, email service skeleton
Phase 7+: Full OIDC authentication, complete email integration, CardDAV contacts

For development, Phase 2+ includes Nextcloud directly in the VoiceAssist docker-compose.yml stack. For production, you may choose to:

Continue using the integrated Nextcloud (simpler deployment)
Use a separate Nextcloud installation (more flexible, as described in the "Separate Stack" section below)

Phase 2: Integrated Nextcloud Setup

What Was Implemented

Phase 2 added Nextcloud directly to the VoiceAssist docker-compose.yml stack for simplified local development:

Docker Services Added:

nextcloud: Nextcloud 29 (Apache), accessible at http://localhost:8080
nextcloud-db: PostgreSQL 16 database for Nextcloud

Integration Service Created:

NextcloudService (services/api-gateway/app/services/nextcloud.py): OCS API client for user provisioning and management

Environment Variables:

NEXTCLOUD_URL=http://nextcloud:80          # Internal Docker network URL
NEXTCLOUD_ADMIN_USER=admin                 # Nextcloud admin username
NEXTCLOUD_ADMIN_PASSWORD=<from .env>       # Nextcloud admin password
NEXTCLOUD_DB_PASSWORD=<from .env>          # Nextcloud database password

OCS API Integration:

User creation via OCS API (/ocs/v1.php/cloud/users)
User existence check
Health check for Nextcloud connectivity
Authentication with admin credentials

Phase 6 Enhancements Implemented:

✅ CalDAV calendar integration (full CRUD operations)
✅ WebDAV file auto-indexing into knowledge base
✅ Email service skeleton (IMAP/SMTP basics)

Future Enhancements (Phase 7+):

OIDC authentication integration
Full email integration with message parsing
CardDAV contacts integration
Full user provisioning workflow

Deployment Steps for OIDC, Contacts, and Email

Configure OIDC providers (API Gateway):
- Set NEXTCLOUD_URL, NEXTCLOUD_OAUTH_CLIENT_ID, NEXTCLOUD_OAUTH_CLIENT_SECRET, and NEXTCLOUD_OAUTH_REDIRECT_URI in .env for Nextcloud SSO.
- Optionally set GOOGLE_OAUTH_CLIENT_ID/SECRET or MICROSOFT_CLIENT_ID/SECRET with their redirect URIs if federating logins.
- Restart the API Gateway; configure_oidc_from_settings() registers providers and caches JWKS for validation.
Enable CardDAV + contacts:
- Provide NEXTCLOUD_WEBDAV_URL, NEXTCLOUD_CALDAV_USERNAME, and NEXTCLOUD_CALDAV_PASSWORD for CardDAV access.
- The CardDAVService now supports sync tokens; call /api/integrations/contacts endpoints to keep address books in sync.
Finalize IMAP/SMTP email:
- Supply EMAIL_IMAP_HOST, EMAIL_IMAP_PORT, EMAIL_SMTP_HOST, EMAIL_SMTP_PORT, EMAIL_USERNAME, and EMAIL_PASSWORD in .env.
- The email service will reconnect automatically on IMAP failures and respects TLS/STARTTLS based on configuration.
Package and install Nextcloud apps:
- Run bash nextcloud-apps/package.sh to create build/*.tar.gz archives for voiceassist-client, voiceassist-admin, and voiceassist-docs.
- Upload/enable the apps in Nextcloud; routes under /apps/<app>/api/* mirror API Gateway endpoints for calendar, files, contacts, and email.

Quick Start (Phase 2)

If you have Phase 2 installed, Nextcloud is already running:

# Access Nextcloud
open http://localhost:8080

# Default credentials (first-time setup)
Username: admin
Password: (value from NEXTCLOUD_ADMIN_PASSWORD in .env)

# Check Nextcloud health from API Gateway
docker exec voiceassist-server python -c "
from app.services.nextcloud import NextcloudService
import asyncio
svc = NextcloudService()
result = asyncio.run(svc.health_check())
print(f'Nextcloud healthy: {result}')
"

Phase 2 Limitations:

OIDC integration not yet implemented (JWT tokens used for auth instead)
WebDAV/CalDAV integration not yet implemented
User provisioning is manual via Nextcloud UI

Phase 6: Calendar & File Integration

What Was Implemented

Phase 6 adds real integration with Nextcloud Calendar and Files, plus email service foundation:

Services Created:

CalDAVService (services/api-gateway/app/services/caldav_service.py): Full CalDAV protocol support for calendar operations
NextcloudFileIndexer (services/api-gateway/app/services/nextcloud_file_indexer.py): Automatic medical document discovery and indexing
EmailService (services/api-gateway/app/services/email_service.py): IMAP/SMTP skeleton for future email integration
Integration API (services/api-gateway/app/api/integrations.py): Unified REST API for all integrations

Calendar Features (CalDAV):

List all calendars for authenticated user
Get events within date range with filtering
Create new calendar events with full metadata
Update existing events (summary, time, location, description)
Delete calendar events
Timezone-aware event handling
Recurring event support
Error handling for connection and parsing failures

File Auto-Indexing (WebDAV):

Discover medical documents in configurable Nextcloud directories
Automatic indexing into Phase 5 knowledge base
Supported formats: PDF, TXT, MD
Duplicate detection (prevents re-indexing)
Metadata tracking (file path, size, modification time)
Integration with Phase 5 KBIndexer for embedding generation
Batch scanning with progress reporting

API Endpoints Added:

Calendar Operations:
  GET    /api/integrations/calendar/calendars
  GET    /api/integrations/calendar/events
  POST   /api/integrations/calendar/events
  PUT    /api/integrations/calendar/events/{uid}
  DELETE /api/integrations/calendar/events/{uid}

File Indexing:
  POST   /api/integrations/files/scan-and-index
  POST   /api/integrations/files/index

Email (Skeleton):
  GET    /api/integrations/email/folders
  GET    /api/integrations/email/messages
  POST   /api/integrations/email/send

Configuration Required:

# Add to ~/VoiceAssist/.env:

# CalDAV Configuration
NEXTCLOUD_CALDAV_URL=http://nextcloud:80/remote.php/dav
NEXTCLOUD_CALDAV_USERNAME=admin
NEXTCLOUD_CALDAV_PASSWORD=<from NEXTCLOUD_ADMIN_PASSWORD>

# WebDAV Configuration
NEXTCLOUD_WEBDAV_URL=http://nextcloud:80/remote.php/dav/files/admin/
NEXTCLOUD_WEBDAV_USERNAME=admin
NEXTCLOUD_WEBDAV_PASSWORD=<from NEXTCLOUD_ADMIN_PASSWORD>

# Watch Directories for Auto-Indexing
NEXTCLOUD_WATCH_DIRECTORIES=/Documents,/Medical_Guidelines

Testing Phase 6 Integrations

Test Calendar Operations:

# List calendars
curl -X GET http://localhost:8000/api/integrations/calendar/calendars \
  -H "Authorization: Bearer $ACCESS_TOKEN"

# Create event
curl -X POST http://localhost:8000/api/integrations/calendar/events \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "summary": "Patient Consultation",
    "start": "2025-01-25T14:00:00Z",
    "end": "2025-01-25T15:00:00Z",
    "description": "Follow-up appointment",
    "location": "Clinic Room 3"
  }'

# Get events in date range
curl -X GET "http://localhost:8000/api/integrations/calendar/events?start_date=2025-01-20&end_date=2025-01-31" \
  -H "Authorization: Bearer $ACCESS_TOKEN"

Test File Auto-Indexing:

# First, add some medical documents to Nextcloud
# Via Nextcloud web UI: Upload PDFs to /Documents folder

# Scan and index all files
curl -X POST "http://localhost:8000/api/integrations/files/scan-and-index?source_type=guideline" \
  -H "Authorization: Bearer $ACCESS_TOKEN"

# Response includes:
# {
#   "files_discovered": 10,
#   "files_indexed": 8,
#   "files_failed": 0,
#   "files_skipped": 2  (already indexed)
# }

# Index specific file
curl -X POST http://localhost:8000/api/integrations/files/index \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "file_path": "/Documents/hypertension_guideline.pdf",
    "source_type": "guideline",
    "title": "2024 Hypertension Management Guidelines"
  }'

Verify Integration:

# Files should now be searchable via Phase 5 RAG
curl -X POST http://localhost:8000/api/realtime/query \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "What are first-line treatments for hypertension?"}'

# Response should include citations from indexed guideline

Phase 6 Limitations

Not Yet Implemented:

OIDC authentication (still using JWT tokens from Phase 2)
Per-user credentials (currently using admin credentials for all operations)
CardDAV contacts integration
Complete email integration (skeleton only)
Calendar event notifications/reminders
Conflict resolution for calendar syncing
Incremental file indexing (currently full scans)

Security Note: Current implementation uses admin credentials for all Nextcloud operations. Production deployments should implement per-user credential management with secure storage (encrypted in database or secrets manager).

Architecture Decision (for Separate Stack Deployment)

Key Principle: Nextcloud and VoiceAssist are independent deployments that communicate via standard APIs.

Separate Stacks:
├── Nextcloud Stack (~/Nextcloud-Dev/)
│   ├── Identity & SSO (OIDC)
│   ├── File Storage (WebDAV)
│   ├── Calendar (CalDAV)
│   ├── Email (IMAP/SMTP)
│   └── User Directory
│
└── VoiceAssist Stack (~/VoiceAssist/)
    ├── Microservices
    ├── Databases
    ├── Observability
    └── Integration with Nextcloud via APIs

Why Separate?

Benefits: ✅ Independence - Update either system without affecting the other ✅ Flexibility - Use existing Nextcloud installation in production ✅ Clarity - Clear separation of concerns ✅ Scalability - Scale each system independently ✅ Maintainability - Easier to troubleshoot and maintain ✅ Reusability - Nextcloud can serve multiple applications

Integration Method:

HTTP/HTTPS APIs (OIDC, WebDAV, CalDAV, CardDAV)
Environment variables for configuration
No shared Docker networks or volumes
No shared databases

Local Development Setup

Directory Structure

~/Nextcloud-Dev/                    # Nextcloud development stack
├── docker-compose.yml              # Nextcloud + Database
├── .env                            # Nextcloud environment
├── data/                           # Nextcloud files
│   ├── data/                       # User files
│   ├── config/                     # Nextcloud config
│   └── apps/                       # Nextcloud apps
└── db/                             # Nextcloud database

~/VoiceAssist/                      # VoiceAssist stack (this repo)
├── docker-compose.yml              # VoiceAssist services
├── .env                            # Includes NEXTCLOUD_* variables
├── services/                       # Microservices
└── data/                           # VoiceAssist data

Step 1: Set Up Nextcloud Dev Stack

Create ~/Nextcloud-Dev directory

mkdir -p ~/Nextcloud-Dev
cd ~/Nextcloud-Dev

Create docker-compose.yml

version: "3.8"

services:
  nextcloud-db:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: nextcloud
      POSTGRES_USER: nextcloud
      POSTGRES_PASSWORD: nextcloud_dev_password
    volumes:
      - nextcloud-db:/var/lib/postgresql/data
    networks:
      - nextcloud-network
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U nextcloud"]
      interval: 10s
      timeout: 5s
      retries: 5

  nextcloud:
    image: nextcloud:latest
    ports:
      - "8080:80"
    environment:
      - POSTGRES_HOST=nextcloud-db
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=nextcloud
      - POSTGRES_PASSWORD=nextcloud_dev_password
      - NEXTCLOUD_ADMIN_USER=admin
      - NEXTCLOUD_ADMIN_PASSWORD=admin_dev_password
      - NEXTCLOUD_TRUSTED_DOMAINS=localhost nextcloud.local
      - OVERWRITEPROTOCOL=http
      - OVERWRITEHOST=localhost:8080
    volumes:
      - nextcloud-data:/var/www/html
    depends_on:
      nextcloud-db:
        condition: service_healthy
    networks:
      - nextcloud-network
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:80/status.php || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3

networks:
  nextcloud-network:
    driver: bridge

volumes:
  nextcloud-db:
    driver: local
  nextcloud-data:
    driver: local

Create .env file

cat > .env <<'EOF'
# Nextcloud Dev Environment

# Database
POSTGRES_DB=nextcloud
POSTGRES_USER=nextcloud
POSTGRES_PASSWORD=nextcloud_dev_password

# Nextcloud Admin
NEXTCLOUD_ADMIN_USER=admin
NEXTCLOUD_ADMIN_PASSWORD=admin_dev_password

# Nextcloud Config
NEXTCLOUD_TRUSTED_DOMAINS=localhost nextcloud.local
EOF

Start Nextcloud Dev Stack

cd ~/Nextcloud-Dev

# Start Nextcloud
docker compose up -d

# Wait for Nextcloud to initialize (first start takes 2-3 minutes)
docker compose logs -f nextcloud

# Check when you see: "Initializing finished"

Access Nextcloud

URL: http://localhost:8080
Username: admin
Password: admin_dev_password

Step 2: Configure Nextcloud for VoiceAssist

Install Required Apps

Access Nextcloud Admin
- Navigate to http://localhost:8080
- Login as admin
- Go to Apps (top right) → Search

Install OIDC App

Search: "OpenID Connect"
Install: "OpenID Connect user backend"

Install External Storage (if not installed)

Search: "External storage support"
Enable if not already enabled

Configure OIDC Provider

Settings → Administration → Security → OAuth 2.0
- Click "Add client"
- Name: VoiceAssist
- Redirection URI: http://localhost:8000/auth/callback
- Type: Confidential
- Click "Add"
- Copy the Client ID and Client Secret - you'll need these

Save Credentials

# Example values (yours will be different):
Client ID: s8dh2k3j4h5k6j7h8k9j0
Client Secret: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

Create Test User

Users → Add User
- Username: testdoc
- Display name: Test Doctor
- Email: testdoc@example.com
- Password: testdoc123
- Groups: users (create if needed)
Verify User Can Login
- Logout as admin
- Login as testdoc
- Verify access works

Step 3: Configure VoiceAssist to Connect to Nextcloud

Update ~/VoiceAssist/.env

Add these variables to your VoiceAssist .env file:

#==============================================
# Nextcloud Integration (Separate Stack)
#==============================================
# Base URL of Nextcloud instance
NEXTCLOUD_BASE_URL=http://localhost:8080

# OIDC Configuration
NEXTCLOUD_OIDC_ISSUER=http://localhost:8080
NEXTCLOUD_CLIENT_ID=s8dh2k3j4h5k6j7h8k9j0        # From Nextcloud OAuth config
NEXTCLOUD_CLIENT_SECRET=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6  # From Nextcloud
NEXTCLOUD_REDIRECT_URI=http://localhost:8000/auth/callback

# WebDAV (for file access)
NEXTCLOUD_WEBDAV_URL=http://localhost:8080/remote.php/dav
NEXTCLOUD_WEBDAV_USERNAME=admin
NEXTCLOUD_WEBDAV_PASSWORD=admin_dev_password

# CalDAV (for calendar integration)
NEXTCLOUD_CALDAV_URL=http://localhost:8080/remote.php/dav/calendars
NEXTCLOUD_CALDAV_USERNAME=admin
NEXTCLOUD_CALDAV_PASSWORD=admin_dev_password

# CardDAV (for contacts)
NEXTCLOUD_CARDDAV_URL=http://localhost:8080/remote.php/dav/addressbooks

# Admin credentials (for service account operations)
NEXTCLOUD_ADMIN_USER=admin
NEXTCLOUD_ADMIN_PASSWORD=admin_dev_password

Step 4: Test Integration

Once VoiceAssist services are running (Phase 2+), test the integration:

# Test 1: Check Nextcloud is reachable
curl http://localhost:8080/status.php
# Should return JSON with Nextcloud status

# Test 2: Test OIDC discovery
curl http://localhost:8080/.well-known/openid-configuration
# Should return OIDC configuration

# Test 3: Test WebDAV access
curl -u admin:admin_dev_password \
  http://localhost:8080/remote.php/dav
# Should return WebDAV capabilities

# Test 4: From VoiceAssist auth service
# (This will be implemented in Phase 2)
docker exec voiceassist-auth-service python -c "
from app.integrations.nextcloud import NextcloudClient
client = NextcloudClient()
print(client.test_connection())
"
# Should print: Connection successful

Production Setup

Assumptions

In production, you likely have:

Existing Nextcloud installation (e.g., https://cloud.asimo.io)
OR need to deploy Nextcloud separately on Ubuntu server
SSL certificates already configured
MFA enabled for all users
Regular backups in place

Integration Steps

1. Identify Nextcloud Instance

# If you have existing Nextcloud:
NEXTCLOUD_BASE_URL=https://cloud.asimo.io

# If deploying fresh Nextcloud on Ubuntu:
# Follow Nextcloud installation guide first
# https://docs.nextcloud.com/server/latest/admin_manual/installation/

2. Configure OIDC in Production Nextcloud

Same steps as local dev, but with production URLs:

Name: VoiceAssist Production
Redirection URI: https://voiceassist.asimo.io/auth/callback
Type: Confidential

3. Update Production VoiceAssist Environment

# In Ubuntu server: ~/VoiceAssist/.env
NEXTCLOUD_BASE_URL=https://cloud.asimo.io
NEXTCLOUD_OIDC_ISSUER=https://cloud.asimo.io
NEXTCLOUD_CLIENT_ID=<production_client_id>
NEXTCLOUD_CLIENT_SECRET=<production_client_secret>
NEXTCLOUD_REDIRECT_URI=https://voiceassist.asimo.io/auth/callback

# Use service account credentials
NEXTCLOUD_ADMIN_USER=voiceassist_service
NEXTCLOUD_ADMIN_PASSWORD=<secure_password>

4. Create Service Account in Nextcloud

Best Practice: Don't use admin account for API access

1. Create user: voiceassist_service
2. Add to group: voiceassist_services
3. Grant necessary permissions:
   - WebDAV access
   - CalDAV access
   - User provisioning (if needed)
4. Generate app password for this account

5. Test Production Integration

# SSH to Ubuntu server
ssh user@asimo.io

# Test Nextcloud connectivity
curl https://cloud.asimo.io/status.php

# Test from VoiceAssist
docker exec voiceassist-auth-service \
  python /app/scripts/test_nextcloud.py

Integration Features

1. Authentication (OIDC)

VoiceAssist Auth Service implements OAuth 2.0 / OIDC client:

# services/auth-service/app/integrations/nextcloud_oidc.py

from authlib.integrations.starlette_client import OAuth

oauth = OAuth()
oauth.register(
    name='nextcloud',
    client_id=settings.NEXTCLOUD_CLIENT_ID,
    client_secret=settings.NEXTCLOUD_CLIENT_SECRET,
    server_metadata_url=f'{settings.NEXTCLOUD_OIDC_ISSUER}/.well-known/openid-configuration',
    client_kwargs={'scope': 'openid profile email'}
)

@app.get('/auth/login')
async def login(request: Request):
    redirect_uri = settings.NEXTCLOUD_REDIRECT_URI
    return await oauth.nextcloud.authorize_redirect(request, redirect_uri)

@app.get('/auth/callback')
async def auth_callback(request: Request):
    token = await oauth.nextcloud.authorize_access_token(request)
    user_info = token.get('userinfo')
    # Create/update user in VoiceAssist database
    # Generate VoiceAssist JWT token
    # Return to client

2. File Storage (WebDAV)

File Indexer Service accesses Nextcloud files:

# services/file-indexer/app/integrations/nextcloud_webdav.py

from webdav3.client import Client

class NextcloudFileClient:
    def __init__(self):
        self.client = Client({
            'webdav_hostname': settings.NEXTCLOUD_WEBDAV_URL,
            'webdav_login': settings.NEXTCLOUD_WEBDAV_USERNAME,
            'webdav_password': settings.NEXTCLOUD_WEBDAV_PASSWORD
        })

    def list_files(self, path='/'):
        return self.client.list(path)

    def download_file(self, remote_path, local_path):
        self.client.download_sync(
            remote_path=remote_path,
            local_path=local_path
        )

    def upload_file(self, local_path, remote_path):
        self.client.upload_sync(
            remote_path=remote_path,
            local_path=local_path
        )

3. Calendar (CalDAV)

Calendar Service integrates with Nextcloud calendars:

# services/calendar-email/app/integrations/nextcloud_caldav.py

import caldav
from datetime import datetime

class NextcloudCalendarClient:
    def __init__(self):
        self.client = caldav.DAVClient(
            url=settings.NEXTCLOUD_CALDAV_URL,
            username=settings.NEXTCLOUD_CALDAV_USERNAME,
            password=settings.NEXTCLOUD_CALDAV_PASSWORD
        )
        self.principal = self.client.principal()

    def get_calendars(self):
        return self.principal.calendars()

    def create_event(self, calendar_name, title, start, end, description=''):
        calendar = self.get_calendar_by_name(calendar_name)
        event = calendar.save_event(
            dtstart=start,
            dtend=end,
            summary=title,
            description=description
        )
        return event

4. Email (IMAP/SMTP or Nextcloud Mail API)

Email Service can use:

Option A: IMAP/SMTP directly
Option B: Nextcloud Mail API (if available)

# services/calendar-email/app/integrations/nextcloud_email.py

import imaplib
import smtplib
from email.mime.text import MIMEText

class NextcloudEmailClient:
    def __init__(self):
        # IMAP for reading
        self.imap = imaplib.IMAP4_SSL('cloud.asimo.io', 993)
        self.imap.login(
            settings.NEXTCLOUD_EMAIL_USERNAME,
            settings.NEXTCLOUD_EMAIL_PASSWORD
        )

        # SMTP for sending
        self.smtp = smtplib.SMTP_SSL('cloud.asimo.io', 465)
        self.smtp.login(
            settings.NEXTCLOUD_EMAIL_USERNAME,
            settings.NEXTCLOUD_EMAIL_PASSWORD
        )

    def fetch_recent_emails(self, mailbox='INBOX', count=10):
        self.imap.select(mailbox)
        _, messages = self.imap.search(None, 'ALL')
        # Fetch and parse emails
        return emails

    def send_email(self, to, subject, body):
        msg = MIMEText(body)
        msg['Subject'] = subject
        msg['From'] = settings.NEXTCLOUD_EMAIL_USERNAME
        msg['To'] = to
        self.smtp.send_message(msg)

Security Considerations

Authentication Security

OIDC Tokens
- Use short-lived access tokens (15 minutes)
- Implement refresh tokens
- Store tokens securely (encrypted in Redis)
- Validate tokens on every request
API Credentials
- Use app passwords instead of user passwords
- Rotate credentials regularly
- Store in environment variables, not code
- Use separate service account for API access

Network Security

Local Development:

HTTP is acceptable for localhost
Consider self-signed certs for HTTPS practice

Production:

ALWAYS use HTTPS for Nextcloud communication
Validate SSL certificates
Use certificate pinning for critical operations
Implement request signing for sensitive operations

Data Privacy

PHI Considerations
- Medical notes in Nextcloud must be encrypted
- Use Nextcloud's encryption module
- Never log file contents
- Implement access controls in Nextcloud
Audit Logging
- Log all Nextcloud API access
- Track file downloads/uploads
- Monitor authentication attempts
- Alert on suspicious activity

Troubleshooting

Nextcloud Connection Issues

# Test 1: Can VoiceAssist reach Nextcloud?
docker exec voiceassist-auth-service \
  curl http://localhost:8080/status.php

# Test 2: Check OIDC configuration
curl http://localhost:8080/.well-known/openid-configuration

# Test 3: Verify OAuth client exists
# Login to Nextcloud → Settings → Administration → Security → OAuth 2.0
# Should see VoiceAssist client

# Test 4: Check redirect URI matches
# In Nextcloud OAuth config: http://localhost:8000/auth/callback
# In VoiceAssist .env: NEXTCLOUD_REDIRECT_URI=http://localhost:8000/auth/callback

Authentication Failures

Problem: "Invalid redirect URI"

Solution: Ensure NEXTCLOUD_REDIRECT_URI in .env matches exactly
what's configured in Nextcloud OAuth client

Problem: "Client authentication failed"

Solution: Verify CLIENT_ID and CLIENT_SECRET are correct
Check for typos, extra spaces

Problem: "Token validation failed"

Solution: Ensure NEXTCLOUD_OIDC_ISSUER is correct
Check that Nextcloud OIDC app is enabled

WebDAV Issues

Problem: "Unauthorized" when accessing files

Solution: Verify NEXTCLOUD_WEBDAV_USERNAME and PASSWORD
Check that user has permission to access files

Problem: "Method not allowed"

Solution: Ensure URL is correct: /remote.php/dav
Not /webdav or /dav

CalDAV Issues

Problem: "Calendar not found"

Solution: Verify calendar exists for the user
Check CALDAV_URL includes /calendars/username/calendar-name

Maintenance

Updating Nextcloud Dev Stack

cd ~/Nextcloud-Dev

# Pull latest image
docker compose pull

# Restart with new image
docker compose down
docker compose up -d

# Check logs
docker compose logs -f nextcloud

Backing Up Nextcloud Dev Data

# Backup volumes
docker run --rm \
  -v nextcloud-dev_nextcloud-data:/data \
  -v ~/Nextcloud-Dev/backups:/backup \
  ubuntu tar czf /backup/nextcloud-data-$(date +%Y%m%d).tar.gz /data

docker run --rm \
  -v nextcloud-dev_nextcloud-db:/data \
  -v ~/Nextcloud-Dev/backups:/backup \
  ubuntu tar czf /backup/nextcloud-db-$(date +%Y%m%d).tar.gz /data

Cleaning Up

cd ~/Nextcloud-Dev

# Stop and remove containers
docker compose down

# Remove volumes (WARNING: deletes all data)
docker compose down -v

# Start fresh
docker compose up -d

Summary

Local Development Checklist

Production Deployment Checklist

Integration Status

Track which integrations are implemented:

Phase 6 Deliverables (Completed):

✅ CalDAV Service with full event CRUD operations
✅ Nextcloud File Indexer for automatic KB population
✅ Email Service skeleton (IMAP/SMTP basics)
✅ Integration API endpoints (/api/integrations/*)
✅ Comprehensive integration tests with mocks
✅ Documentation updates

Deferred to Phase 7+:

⏳ OIDC authentication flow
⏳ Per-user credential management
⏳ CardDAV contacts integration
⏳ Full email integration (parsing, threading, search)
⏳ Calendar notifications and reminders
⏳ Incremental file indexing with change detection

References

VoiceAssist V2 - Tools and Integrations

Last Updated: 2025-11-20 Status: Design Complete Version: V2.0

Implementation Note: This document describes the V2 tools architecture design. The current production implementation lives in:

Tool Service: services/api-gateway/app/services/tools/tool_service.py

Individual Tools: services/api-gateway/app/services/tools/*.py

Legacy Location: server/app/tools/ (deprecated)

Overview
Tool Architecture
Tool Registry
Tool Definitions
Tool Security Model
Tool Invocation Flow
Tool Results and Citations
Frontend Integration
Observability and Monitoring
Error Handling
Testing Tools
Future Tools

Overview

VoiceAssist V2 implements a first-class tools layer that allows the AI model to take actions on behalf of the user. Tools are integrated with the OpenAI Realtime API and the backend orchestrator to provide:

Calendar operations (view events, create appointments)
File operations (search Nextcloud files, retrieve documents)
Medical knowledge retrieval (OpenEvidence, PubMed, guidelines)
Medical calculations (dosing, risk scores, differential diagnosis)
Web search (current medical information)

Key Features

Type-Safe: All tools use Pydantic models for arguments and results
PHI-Aware: Tools classified by PHI handling capability
Auditable: All tool calls logged with full parameters and results
Secure: Permission checks, rate limiting, input validation
Observable: Prometheus metrics for all tool invocations
User-Confirmed: High-risk tools require user confirmation before execution

Design Principles

Least Privilege: Tools only get access they need
Explicit > Implicit: Always require user confirmation for risky operations
Fail Safe: Errors should not expose PHI or system internals
Auditable: Every tool call recorded with full context
Testable: Mock implementations for testing

Tool Architecture

Components

┌─────────────────────────────────────────────────────────────────┐
│                        OpenAI Realtime API                      │
│                  (Function Calling / Tools)                     │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                       Voice Proxy Service                       │
│  - Receives tool calls from OpenAI                              │
│  - Validates tool arguments                                     │
│  - Routes to backend orchestrator                               │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Query Orchestrator Service                   │
│  - Tool execution engine                                        │
│  - PHI detection and routing                                    │
│  - Confirmation management                                      │
│  - Result assembly                                              │
└────────────────────────────┬────────────────────────────────────┘
                             │
                  ┌──────────┴──────────┐
                  ▼                     ▼
┌─────────────────────────┐   ┌────────────────────────┐
│   Local Tool Modules    │   │  External API Clients  │
│  - calendar_tool.py     │   │  - OpenEvidence        │
│  - nextcloud_tool.py    │   │  - PubMed              │
│  - calculator_tool.py   │   │  - Web Search          │
│  - diagnosis_tool.py    │   │  - UpToDate (if avail) │
└─────────────────────────┘   └────────────────────────┘

Tool Lifecycle

Registration: Tool defined with schema, registered in TOOL_REGISTRY
Discovery: OpenAI Realtime API receives tool definitions
Invocation: AI model calls tool with arguments
Validation: Arguments validated against Pydantic model
Authorization: Permission check (user, PHI status, rate limits)
Confirmation: User prompt if required (high-risk operations)
Execution: Tool logic runs (API call, calculation, file access)
Result: Structured result returned to AI model
Audit: Tool call logged to database with full context
Citation: Result converted to citation if needed

Tool Registry

TOOL_REGISTRY Structure

All tools are registered in server/app/tools/registry.py:

from typing import Dict, Type, Callable
from pydantic import BaseModel
from app.tools.base import ToolDefinition, ToolResult

TOOL_REGISTRY: Dict[str, ToolDefinition] = {}
TOOL_MODELS: Dict[str, Type[BaseModel]] = {}
TOOL_HANDLERS: Dict[str, Callable] = {}

def register_tool(
    name: str,
    definition: ToolDefinition,
    model: Type[BaseModel],
    handler: Callable
):
    """Register a tool with schema, model, and handler"""
    TOOL_REGISTRY[name] = definition
    TOOL_MODELS[name] = model
    TOOL_HANDLERS[name] = handler

ToolDefinition Schema

class ToolDefinition(BaseModel):
    """Tool definition for OpenAI Realtime API"""
    name: str
    description: str
    parameters: Dict[str, Any]  # JSON Schema

    # VoiceAssist-specific metadata
    category: str  # "calendar", "file", "medical", "calculation", "search"
    requires_phi: bool  # True if tool processes PHI
    requires_confirmation: bool  # True if user must confirm
    risk_level: str  # "low", "medium", "high"
    rate_limit: Optional[int] = None  # Max calls per minute
    timeout_seconds: int = 30

Tool Definitions

Tool 1: Get Calendar Events

Purpose: Retrieve calendar events for a date range

Tool Name: get_calendar_events

Category: Calendar

PHI Status: requires_phi: true (patient appointments may contain PHI)

Confirmation Required: false (read-only)

Risk Level: low

Arguments:

class GetCalendarEventsArgs(BaseModel):
    start_date: str  # ISO 8601 format (YYYY-MM-DD)
    end_date: str    # ISO 8601 format (YYYY-MM-DD)
    calendar_name: Optional[str] = None  # Filter by calendar
    max_results: Optional[int] = 50

Returns:

class CalendarEvent(BaseModel):
    id: str
    title: str
    start: str  # ISO 8601 datetime
    end: str    # ISO 8601 datetime
    location: Optional[str] = None
    description: Optional[str] = None
    calendar_name: str
    all_day: bool = False

class GetCalendarEventsResult(BaseModel):
    events: List[CalendarEvent]
    total_count: int
    date_range: str  # "2024-01-15 to 2024-01-20"

Example Call:

{
  "tool": "get_calendar_events",
  "arguments": {
    "start_date": "2024-01-15",
    "end_date": "2024-01-20"
  }
}

Implementation: CalDAV integration with Nextcloud Calendar

Tool 2: Create Calendar Event

Purpose: Create a new calendar event

Tool Name: create_calendar_event

Category: Calendar

PHI Status: requires_phi: true (may contain patient names)

Confirmation Required: true (creates data)

Risk Level: medium

Arguments:

class CreateCalendarEventArgs(BaseModel):
    title: str
    start_datetime: str  # ISO 8601 datetime
    end_datetime: str    # ISO 8601 datetime
    location: Optional[str] = None
    description: Optional[str] = None
    calendar_name: Optional[str] = "Default"
    all_day: bool = False

Returns:

class CreateCalendarEventResult(BaseModel):
    event_id: str
    title: str
    start: str
    end: str
    created: bool = True
    message: str  # "Event created successfully"

Confirmation Prompt:

"I'd like to create a calendar event:
- Title: {title}
- Start: {start_datetime}
- End: {end_datetime}
- Location: {location}

Should I proceed?"

Implementation: CalDAV POST to Nextcloud Calendar

Tool 3: Search Nextcloud Files

Purpose: Search for files in Nextcloud by name or content

Tool Name: search_nextcloud_files

Category: File

PHI Status: requires_phi: true (files may contain PHI)

Confirmation Required: false (read-only)

Risk Level: low

Arguments:

class SearchNextcloudFilesArgs(BaseModel):
    query: str  # Search query
    file_type: Optional[str] = None  # "pdf", "docx", "txt", etc.
    max_results: Optional[int] = 20
    include_content: bool = False  # Search file contents

Returns:

class NextcloudFile(BaseModel):
    file_id: str
    name: str
    path: str
    size: int  # bytes
    mime_type: str
    modified: str  # ISO 8601 datetime
    url: str  # WebDAV URL

class SearchNextcloudFilesResult(BaseModel):
    files: List[NextcloudFile]
    total_count: int
    query: str

Example Call:

{
  "tool": "search_nextcloud_files",
  "arguments": {
    "query": "diabetes guidelines",
    "file_type": "pdf",
    "max_results": 10
  }
}

Implementation: Nextcloud WebDAV search API

Tool 4: Retrieve Nextcloud File

Purpose: Retrieve the contents of a specific Nextcloud file

Tool Name: retrieve_nextcloud_file

Category: File

PHI Status: requires_phi: true (file may contain PHI)

Confirmation Required: false (read-only)

Risk Level: low

Arguments:

class RetrieveNextcloudFileArgs(BaseModel):
    file_id: str  # File ID from search results
    extract_text: bool = True  # Extract text from PDF/DOCX
    max_chars: Optional[int] = 10000  # Limit text extraction

Returns:

class RetrieveNextcloudFileResult(BaseModel):
    file_id: str
    name: str
    content: Optional[str] = None  # Extracted text
    content_truncated: bool = False
    mime_type: str
    size: int
    url: str

Implementation: WebDAV GET + text extraction (PyPDF2, docx)

Tool 5: Search OpenEvidence

Purpose: Search OpenEvidence API for medical evidence

Tool Name: search_openevidence

Category: Medical

PHI Status: requires_phi: false (external API, no PHI sent)

Confirmation Required: false (read-only external API)

Risk Level: low

Arguments:

class SearchOpenEvidenceArgs(BaseModel):
    query: str  # Medical question
    max_results: Optional[int] = 5
    evidence_level: Optional[str] = None  # "high", "moderate", "low"

Returns:

class OpenEvidenceResult(BaseModel):
    title: str
    summary: str
    evidence_level: str  # "high", "moderate", "low"
    source: str  # Journal name
    pubmed_id: Optional[str] = None
    url: str
    date: Optional[str] = None

class SearchOpenEvidenceResponse(BaseModel):
    results: List[OpenEvidenceResult]
    total_count: int
    query: str

Example Call:

{
  "tool": "search_openevidence",
  "arguments": {
    "query": "beta blockers in heart failure",
    "max_results": 5
  }
}

Implementation: OpenEvidence REST API client

Tool 6: Search PubMed

Purpose: Search PubMed for medical literature

Tool Name: search_pubmed

Category: Medical

PHI Status: requires_phi: false (external API, no PHI sent)

Confirmation Required: false (read-only external API)

Risk Level: low

Arguments:

class SearchPubMedArgs(BaseModel):
    query: str  # PubMed search query
    max_results: Optional[int] = 10
    publication_types: Optional[List[str]] = None  # ["Clinical Trial", "Review"]
    date_from: Optional[str] = None  # YYYY/MM/DD
    date_to: Optional[str] = None    # YYYY/MM/DD

Returns:

class PubMedArticle(BaseModel):
    pmid: str
    title: str
    authors: List[str]
    journal: str
    publication_date: str
    abstract: Optional[str] = None
    doi: Optional[str] = None
    url: str  # PubMed URL

class SearchPubMedResult(BaseModel):
    articles: List[PubMedArticle]
    total_count: int
    query: str

Implementation: NCBI E-utilities API (esearch + efetch)

Tool 7: Calculate Medical Score

Purpose: Calculate medical risk scores and dosing

Tool Name: calculate_medical_score

Category: Calculation

PHI Status: requires_phi: true (patient data used in calculation)

Confirmation Required: false (deterministic calculation)

Risk Level: medium (results used for clinical decisions)

Arguments:

class CalculateMedicalScoreArgs(BaseModel):
    calculator_name: str  # "wells_dvt", "chadsvasc", "grace", "renal_dosing"
    parameters: Dict[str, Any]  # Calculator-specific parameters

Returns:

class MedicalScoreResult(BaseModel):
    calculator_name: str
    score: Union[float, str]
    interpretation: str
    risk_category: Optional[str] = None
    recommendations: Optional[List[str]] = None
    parameters_used: Dict[str, Any]

Example Call (Wells' DVT Score):

{
  "tool": "calculate_medical_score",
  "arguments": {
    "calculator_name": "wells_dvt",
    "parameters": {
      "active_cancer": true,
      "paralysis_recent": false,
      "bedridden_3days": true,
      "localized_tenderness": true,
      "entire_leg_swollen": false,
      "calf_swelling_3cm": true,
      "pitting_edema": true,
      "collateral_veins": false,
      "alternative_diagnosis": false
    }
  }
}

Example Result:

{
  "calculator_name": "wells_dvt",
  "score": 6.0,
  "interpretation": "High probability of DVT",
  "risk_category": "high",
  "recommendations": [
    "Consider urgent ultrasound",
    "Consider empiric anticoagulation if no contraindications"
  ],
  "parameters_used": { ... }
}

Supported Calculators:

wells_dvt: Wells' Criteria for DVT
wells_pe: Wells' Criteria for Pulmonary Embolism
chadsvasc: CHA2DS2-VASc Score (stroke risk in AFib)
hasbled: HAS-BLED Score (bleeding risk)
grace: GRACE Score (ACS risk)
meld: MELD Score (liver disease severity)
renal_dosing: Renal dose adjustment calculator
bmi: BMI calculator with interpretation

Implementation: Local calculation library (no external API)

Tool 8: Search Medical Guidelines

Purpose: Search curated medical guidelines (CDC, WHO, specialty societies)

Tool Name: search_medical_guidelines

Category: Medical

PHI Status: requires_phi: false (local database search)

Confirmation Required: false (read-only)

Risk Level: low

Arguments:

class SearchMedicalGuidelinesArgs(BaseModel):
    query: str  # Search query
    guideline_source: Optional[str] = None  # "cdc", "who", "acc", "aha", etc.
    condition: Optional[str] = None  # Filter by condition
    max_results: Optional[int] = 10

Returns:

class MedicalGuideline(BaseModel):
    id: str
    title: str
    source: str  # "CDC", "WHO", "AHA", etc.
    condition: str  # "Diabetes", "Hypertension", etc.
    summary: str
    url: str
    publication_date: str
    last_updated: str

class SearchMedicalGuidelinesResult(BaseModel):
    guidelines: List[MedicalGuideline]
    total_count: int
    query: str

Implementation: Local vector search in knowledge base (guidelines ingested via automated scrapers)

Tool 9: Generate Differential Diagnosis

Purpose: Generate differential diagnosis list based on symptoms

Tool Name: generate_differential_diagnosis

Category: Medical

PHI Status: requires_phi: true (patient symptoms)

Confirmation Required: false (informational only)

Risk Level: medium (clinical decision support)

Arguments:

class GenerateDifferentialDiagnosisArgs(BaseModel):
    chief_complaint: str
    symptoms: List[str]
    patient_age: Optional[int] = None
    patient_sex: Optional[str] = None  # "M", "F", "Other"
    relevant_history: Optional[List[str]] = None
    max_results: Optional[int] = 10

Returns:

class DiagnosisCandidate(BaseModel):
    diagnosis: str
    probability: str  # "high", "medium", "low"
    key_features: List[str]  # Matching symptoms
    missing_features: List[str]  # Expected but not present
    next_steps: List[str]  # Recommended workup

class GenerateDifferentialDiagnosisResult(BaseModel):
    chief_complaint: str
    diagnoses: List[DiagnosisCandidate]
    reasoning: str
    disclaimers: List[str] = [
        "This is not a substitute for clinical judgment",
        "Consider patient context and physical exam findings"
    ]

Example Call:

{
  "tool": "generate_differential_diagnosis",
  "arguments": {
    "chief_complaint": "chest pain",
    "symptoms": ["substernal chest pressure", "shortness of breath", "diaphoresis", "pain radiating to left arm"],
    "patient_age": 65,
    "patient_sex": "M",
    "relevant_history": ["hypertension", "diabetes", "smoking history"],
    "max_results": 5
  }
}

Implementation: RAG system + medical knowledge base + BioGPT

Tool 10: Web Search (Medical)

Purpose: Search the web for current medical information

Tool Name: web_search_medical

Category: Search

PHI Status: requires_phi: false (no PHI sent to external search)

Confirmation Required: false (read-only)

Risk Level: low

Arguments:

class WebSearchMedicalArgs(BaseModel):
    query: str
    max_results: Optional[int] = 5
    domain_filter: Optional[List[str]] = None  # ["nih.gov", "cdc.gov"]

Returns:

class WebSearchResult(BaseModel):
    title: str
    url: str
    snippet: str
    domain: str
    date: Optional[str] = None

class WebSearchMedicalResponse(BaseModel):
    results: List[WebSearchResult]
    total_count: int
    query: str

Implementation: Google Custom Search API or Brave Search API (filtered to medical domains)

Tool Security Model

PHI Handling Rules

Tools with requires_phi: true:

Calendar tools (patient appointments)
Nextcloud file tools (may contain patient documents)
Medical calculators (patient data)
Differential diagnosis (patient symptoms)

Security Requirements:

Must not send PHI to external APIs
Must run locally or use HIPAA-compliant services
Must log tool calls with PHI flag
Must redact PHI from error messages

Tools with requires_phi: false:

OpenEvidence search
PubMed search
Medical guidelines search
Web search

Security Requirements:

Safe to call external APIs
No PHI in query parameters
Rate limiting to prevent abuse

Confirmation Requirements

Tools requiring user confirmation (requires_confirmation: true):

create_calendar_event: Creates data
Any tool that modifies state

Confirmation Flow:

Tool call received from OpenAI
Orchestrator detects requires_confirmation: true
Send confirmation request to frontend
User approves/denies via UI
If approved, execute tool and return result
If denied, return "User declined" message to AI

Rate Limiting

Per-Tool Rate Limits:

Calendar tools: 10 calls/minute
File tools: 20 calls/minute
Medical search tools: 30 calls/minute
Calculators: 50 calls/minute (local, fast)

Implementation: Redis-based rate limiter with sliding window

Input Validation

All tool arguments validated with Pydantic:

Type checking
Range validation
Format validation (dates, enums)
Length limits (prevent injection attacks)

Example Validation:

class GetCalendarEventsArgs(BaseModel):
    start_date: str = Field(..., regex=r'^\d{4}-\d{2}-\d{2}$')
    end_date: str = Field(..., regex=r'^\d{4}-\d{2}-\d{2}$')
    max_results: Optional[int] = Field(50, ge=1, le=100)

Tool Invocation Flow

Step-by-Step Flow

1. User speaks: "What's on my calendar tomorrow?"
   ↓
2. OpenAI Realtime API recognizes need for tool call
   ↓
3. OpenAI calls tool: get_calendar_events(start_date="2024-01-16", end_date="2024-01-16")
   ↓
4. Voice Proxy receives tool call
   ↓
5. Voice Proxy forwards to Query Orchestrator
   ↓
6. Orchestrator validates arguments (Pydantic)
   ↓
7. Orchestrator checks permissions (user auth, rate limit)
   ↓
8. Orchestrator detects PHI: true (calendar events)
   ↓
9. Orchestrator checks confirmation: false (read-only)
   ↓
10. Orchestrator executes tool handler: calendar_tool.get_events()
    ↓
11. Tool handler calls CalDAV API → Nextcloud
    ↓
12. Nextcloud returns events
    ↓
13. Tool handler returns structured result
    ↓
14. Orchestrator logs tool call to audit log
    ↓
15. Orchestrator creates ToolResult object
    ↓
16. ToolResult returned to Voice Proxy
    ↓
17. Voice Proxy sends result to OpenAI Realtime API
    ↓
18. OpenAI synthesizes natural language response: "You have 3 meetings tomorrow..."
    ↓
19. User hears response

Confirmation Flow (for high-risk tools)

1. User: "Create a meeting with Dr. Smith at 2pm tomorrow"
   ↓
2. OpenAI calls: create_calendar_event(title="Meeting with Dr. Smith", ...)
   ↓
3. Orchestrator detects requires_confirmation: true
   ↓
4. Orchestrator sends confirmation request to frontend:
   {
     "type": "tool_confirmation",
     "tool": "create_calendar_event",
     "arguments": { ... },
     "prompt": "I'd like to create a calendar event: ..."
   }
   ↓
5. Frontend displays confirmation dialog
   ↓
6. User clicks "Confirm" or "Cancel"
   ↓
7. Frontend sends confirmation response
   ↓
8. If confirmed:
   - Orchestrator executes tool
   - Returns result to OpenAI
   ↓
9. If cancelled:
   - Orchestrator returns "User declined"
   - OpenAI acknowledges: "Okay, I won't create that event"

Tool Results and Citations

ToolResult Structure

class ToolResult(BaseModel):
    tool_name: str
    success: bool
    result: Optional[Dict[str, Any]] = None
    error: Optional[str] = None
    execution_time_ms: float
    timestamp: str  # ISO 8601

    # Citation metadata (if applicable)
    citations: Optional[List[Citation]] = None

Converting Tool Results to Citations

For tools that return citable content:

def tool_result_to_citation(tool_result: ToolResult) -> List[Citation]:
    """Convert tool result to citations"""
    if tool_result.tool_name == "search_openevidence":
        return [
            Citation(
                text=result["summary"],
                source_type="openevidence",
                source_title=result["title"],
                source_url=result["url"],
                relevance_score=0.95,
                metadata={
                    "evidence_level": result["evidence_level"],
                    "pubmed_id": result.get("pubmed_id")
                }
            )
            for result in tool_result.result["results"]
        ]
    # ... other tools

Citation Display in Frontend:

Show tool name as citation source
Link to external URLs if available
Display metadata (e.g., evidence level, date)

Frontend Integration

React Hooks for Tools

// web-app/src/hooks/useToolConfirmation.ts
export function useToolConfirmation() {
  const [pendingTool, setPendingTool] = useState<ToolConfirmation | null>(null);

  const handleToolConfirmation = (confirmation: ToolConfirmation) => {
    setPendingTool(confirmation);
  };

  const confirmTool = async () => {
    // Send confirmation to backend
    await api.confirmTool(pendingTool.tool_call_id, true);
    setPendingTool(null);
  };

  const cancelTool = async () => {
    await api.confirmTool(pendingTool.tool_call_id, false);
    setPendingTool(null);
  };

  return { pendingTool, confirmTool, cancelTool };
}

Tool Confirmation Dialog

// web-app/src/components/ToolConfirmationDialog.tsx
export function ToolConfirmationDialog({
  toolCall,
  onConfirm,
  onCancel
}: ToolConfirmationProps) {
  return (
    <Dialog open={!!toolCall}>
      <DialogTitle>Confirm Action</DialogTitle>
      <DialogContent>
        <p>{toolCall.confirmation_prompt}</p>

        <div className="tool-details">
          <h4>Details:</h4>
          <pre>{JSON.stringify(toolCall.arguments, null, 2)}</pre>
        </div>
      </DialogContent>
      <DialogActions>
        <Button onClick={onCancel} variant="outlined">
          Cancel
        </Button>
        <Button onClick={onConfirm} variant="contained">
          Confirm
        </Button>
      </DialogActions>
    </Dialog>
  );
}

Tool Activity Indicator

// Show when tools are running
export function ToolActivityIndicator({ activeTool }: { activeTool: string | null }) {
  if (!activeTool) return null;

  return (
    <div className="tool-activity">
      <CircularProgress size={16} />
      <span>Running tool: {activeTool}</span>
    </div>
  );
}

Observability and Monitoring

Prometheus Metrics

Tool Invocation Metrics:

tool_calls_total = Counter(
    'voiceassist_tool_calls_total',
    'Total number of tool calls',
    ['tool_name', 'status']  # status: success, error, denied
)

tool_execution_duration_seconds = Histogram(
    'voiceassist_tool_execution_duration_seconds',
    'Tool execution duration',
    ['tool_name']
)

tool_confirmation_rate = Gauge(
    'voiceassist_tool_confirmation_rate',
    'Percentage of tool calls confirmed by users',
    ['tool_name']
)

tool_error_rate = Gauge(
    'voiceassist_tool_error_rate',
    'Tool error rate (errors / total calls)',
    ['tool_name']
)

Example Dashboard Panel:

Tool Call Rate (calls/minute)
- get_calendar_events: 15/min
- search_pubmed: 8/min
- calculate_medical_score: 5/min
- create_calendar_event: 2/min

Tool Success Rate
- get_calendar_events: 99.5%
- search_pubmed: 98.2%
- calculate_medical_score: 100%
- create_calendar_event: 95.0% (5% denied by users)

Tool P95 Latency
- get_calendar_events: 250ms
- search_pubmed: 1200ms
- search_openevidence: 1500ms
- calculate_medical_score: 50ms

Structured Logging

logger.info(
    "tool_call",
    extra={
        "tool_name": "get_calendar_events",
        "user_id": 123,
        "session_id": "abc123",
        "arguments": {"start_date": "2024-01-15", "end_date": "2024-01-20"},
        "execution_time_ms": 245,
        "status": "success",
        "phi_detected": True,
        "confirmation_required": False
    }
)

PHI Redaction:

Never log file contents
Never log patient names, MRNs
Hash user identifiers
Redact PHI from error messages

Audit Logging

All tool calls logged to audit_logs table:

INSERT INTO audit_logs (
  user_id,
  action_type,
  resource_type,
  resource_id,
  action_details,
  phi_involved,
  ip_address,
  user_agent,
  timestamp
) VALUES (
  123,
  'tool_call',
  'calendar',
  NULL,
  '{"tool": "get_calendar_events", "start_date": "2024-01-15"}',
  TRUE,
  '192.168.1.10',
  'VoiceAssist/2.0',
  NOW()
);

Error Handling

Error Types

1. Validation Errors (400 Bad Request)

{
  "error": "validation_error",
  "message": "Invalid arguments for tool 'get_calendar_events'",
  "details": {
    "start_date": "Invalid date format, expected YYYY-MM-DD"
  }
}

2. Permission Errors (403 Forbidden)

{
  "error": "permission_denied",
  "message": "User does not have permission to call tool 'create_calendar_event'"
}

3. Rate Limit Errors (429 Too Many Requests)

{
  "error": "rate_limit_exceeded",
  "message": "Rate limit exceeded for tool 'search_pubmed'",
  "details": {
    "limit": 30,
    "window": "1 minute",
    "retry_after": 15  # seconds
  }
}

4. External API Errors (502 Bad Gateway)

{
  "error": "external_api_error",
  "message": "Failed to call PubMed API",
  "details": {
    "upstream_status": 503,
    "upstream_message": "Service temporarily unavailable"
  }
}

5. Timeout Errors (504 Gateway Timeout)

{
  "error": "timeout",
  "message": "Tool execution exceeded timeout of 30 seconds"
}

Error Recovery

Retry Strategy:

Transient errors (5xx): Retry up to 3 times with exponential backoff
Rate limits: Wait and retry after retry_after seconds
Validation errors: Do not retry (client error)

Fallback Behavior:

If external API fails, fall back to local knowledge base
If tool times out, return partial results if available
If confirmation denied, inform AI model gracefully

Testing Tools

Unit Tests

# server/tests/tools/test_calendar_tool.py
def test_get_calendar_events():
    args = GetCalendarEventsArgs(
        start_date="2024-01-15",
        end_date="2024-01-20"
    )

    result = calendar_tool.get_events(args, user_id=1)

    assert result.success is True
    assert len(result.result["events"]) > 0
    assert result.execution_time_ms < 1000

Integration Tests

def test_tool_invocation_flow():
    # Simulate OpenAI tool call
    tool_call = {
        "tool": "get_calendar_events",
        "arguments": {
            "start_date": "2024-01-15",
            "end_date": "2024-01-20"
        }
    }

    # Send to orchestrator
    response = orchestrator.execute_tool(tool_call, user_id=1)

    # Verify result
    assert response["success"] is True
    assert "events" in response["result"]

Mock Tools for Testing

# server/app/tools/mocks.py
class MockCalendarTool:
    def get_events(self, args, user_id):
        return ToolResult(
            tool_name="get_calendar_events",
            success=True,
            result={
                "events": [
                    {
                        "id": "mock-1",
                        "title": "Team Meeting",
                        "start": "2024-01-15T10:00:00Z",
                        "end": "2024-01-15T11:00:00Z"
                    }
                ],
                "total_count": 1
            },
            execution_time_ms=50,
            timestamp="2024-01-15T09:00:00Z"
        )

Admin Panel Testing UI

Feature: Test Tool Calls

Dropdown to select tool
Form to enter arguments (JSON editor)
"Execute Tool" button
Display result or error
Show execution time and metrics

Future Tools

Phase 11+ Enhancements

Additional Tools to Consider:

Email Search: search_email - Search Nextcloud Mail
Send Email: send_email - Send email (requires confirmation)
Task Management: create_task, get_tasks - Nextcloud Tasks integration
Drug Interaction Check: check_drug_interactions - Check for drug interactions
Clinical Trial Search: search_clinical_trials - ClinicalTrials.gov API
Lab Value Interpretation: interpret_lab_values - Interpret lab results
ICD-10 Code Lookup: lookup_icd10 - Look up diagnosis codes
Medical Abbreviation Lookup: lookup_medical_abbreviation - Expand medical abbreviations

API Expansion

UpToDate API: If licensed, add search_uptodate tool
FHIR Integration: Tools to query EHR systems via FHIR
HL7 Integration: Parse HL7 messages for data extraction

ORCHESTRATION_DESIGN.md - How tools fit into query orchestration
DATA_MODEL.md - ToolCall and ToolResult entities
SERVICE_CATALOG.md - Tool execution service
SECURITY_COMPLIANCE.md - PHI handling rules
OBSERVABILITY.md - Metrics and logging
WEB_APP_SPECS.md - Frontend tool confirmation UI

Last Updated: 2025-11-20 Version: V2.0 Total Tools: 10 (with 8+ future tools)

Nextcloud Integration Guide

Overview

Phase 2: Integrated Nextcloud Setup

What Was Implemented

Deployment Steps for OIDC, Contacts, and Email

Quick Start (Phase 2)

Phase 6: Calendar & File Integration

What Was Implemented

Testing Phase 6 Integrations

Phase 6 Limitations

Architecture Decision (for Separate Stack Deployment)

Why Separate?

Local Development Setup

Directory Structure

Step 1: Set Up Nextcloud Dev Stack

Create ~/Nextcloud-Dev directory

Create docker-compose.yml

Create .env file

Start Nextcloud Dev Stack

Access Nextcloud

Step 2: Configure Nextcloud for VoiceAssist

Install Required Apps

Configure OIDC Provider

Create Test User

Step 3: Configure VoiceAssist to Connect to Nextcloud

Update ~/VoiceAssist/.env

Step 4: Test Integration

Production Setup

Assumptions

Integration Steps

1. Identify Nextcloud Instance

2. Configure OIDC in Production Nextcloud

3. Update Production VoiceAssist Environment

4. Create Service Account in Nextcloud

5. Test Production Integration

Integration Features

1. Authentication (OIDC)

2. File Storage (WebDAV)

3. Calendar (CalDAV)

4. Email (IMAP/SMTP or Nextcloud Mail API)

Security Considerations

Authentication Security

Network Security

Data Privacy

Troubleshooting

Nextcloud Connection Issues

Authentication Failures

WebDAV Issues

CalDAV Issues

Maintenance

Updating Nextcloud Dev Stack

Backing Up Nextcloud Dev Data

Cleaning Up

Summary

Local Development Checklist

Production Deployment Checklist

Integration Status

References

VoiceAssist V2 - Tools and Integrations

Table of Contents

Overview

Key Features

Design Principles

Tool Architecture

Components

Tool Lifecycle

Tool Registry

TOOL_REGISTRY Structure

ToolDefinition Schema

Tool Definitions

Tool 1: Get Calendar Events

Tool 2: Create Calendar Event

Tool 3: Search Nextcloud Files

Tool 4: Retrieve Nextcloud File

Tool 5: Search OpenEvidence

Tool 6: Search PubMed

Tool 7: Calculate Medical Score

Tool 8: Search Medical Guidelines

Tool 9: Generate Differential Diagnosis

Tool 10: Web Search (Medical)