Contract Testing Guide

Last Updated: 2025-11-21 (Phase 7 - P3.4) Purpose: Guide for implementing and running contract tests with Pact

Overview

VoiceAssist V2 uses Pact for consumer-driven contract testing. Contract tests ensure that the API Gateway meets the expectations of all consumers (frontend, mobile apps) without requiring integration tests with real services.

Benefits:

• Catch breaking API changes before deployment
• Test independently of consumer/provider availability
• Document API contracts in executable form
• Enable parallel development of frontend and backend

Architecture

┌─────────────────────┐
│  Consumer (Frontend)│
│  Defines Contract   │
└──────────┬──────────┘
           │ Pact File
           ▼
┌─────────────────────┐
│   Pact Broker       │
│  Stores Contracts   │
└──────────┬──────────┘
           │ Verification
           ▼
┌─────────────────────┐
│ Provider (API GW)   │
│ Verifies Contract   │
└─────────────────────┘

Components:

1. Consumer Tests: Frontend defines what it expects from API
2. Pact Files: JSON contracts generated by consumer tests
3. Pact Broker: Central repository for contracts
4. Provider Tests: API Gateway verifies it meets contracts

Setup

1. Install Dependencies

Already added to requirements.txt:

pact-python==2.2.0

Rebuild containers:

docker compose build voiceassist-server

2. Start Pact Broker

# Start Pact Broker (already in docker-compose.yml)
docker compose up -d pact-broker

# Verify broker is running
curl http://localhost:9292/diagnostic/status/heartbeat

# Access UI
open http://localhost:9292
# Login: pact / pact

3. Database Setup

Pact Broker uses PostgreSQL. Create database:

docker compose exec postgres psql -U voiceassist -c "CREATE DATABASE pact_broker;"

Writing Consumer Tests

Consumer tests define what the frontend expects from the API.

Example: Auth Login Contract

# tests/contract/test_auth_contract.py

from pact import Consumer, Provider, Like

@pytest.fixture(scope="session")
def pact(request):
    pact = Consumer("VoiceAssistFrontend").has_pact_with(
        Provider("VoiceAssistAPI"),
        host_name="localhost",
        port=1234,  # Mock server port
        pact_dir="./pacts"
    )
    pact.start_service()
    yield pact
    pact.stop_service()

def test_login_success_contract(self, pact):
    """Contract: POST /api/auth/login returns tokens."""
    expected_response = {
        "access_token": Like("eyJ..."),  # Match any string
        "refresh_token": Like("eyJ..."),
        "token_type": "bearer",
        "expires_in": Like(900)  # Match any number
    }

    (pact
     .given("user exists with valid credentials")  # Provider state
     .upon_receiving("a request to login")         # Interaction name
     .with_request(
         method="POST",
         path="/api/auth/login",
         body={"email": "test@example.com", "password": "testpass"}
     )
     .will_respond_with(200, body=expected_response))

    with pact:
        # Make request to mock server
        response = requests.post(f"{pact.uri}/api/auth/login", ...)
        assert response.status_code == 200

Run Consumer Tests

# Run consumer tests - generates pact files
pytest tests/contract/test_auth_contract.py -k Consumer

# Pact files created in: ./pacts/
ls pacts/
# voiceassistfrontend-voiceassistapi.json

Publish to Broker

# Publish pact to broker
pact-broker publish \
  pacts/voiceassistfrontend-voiceassistapi.json \
  --consumer-app-version=1.0.0 \
  --broker-base-url=http://localhost:9292 \
  --broker-username=pact \
  --broker-password=pact

Writing Provider Tests

Provider tests verify the actual API meets contracts.

Example: Verify Auth Contract

def test_verify_pact_with_provider():
    """Verify API Gateway meets contract."""
    from pact import Verifier

    verifier = Verifier(
        provider="VoiceAssistAPI",
        provider_base_url="http://localhost:8000"
    )

    success, logs = verifier.verify_pacts(
        "./pacts/voiceassistfrontend-voiceassistapi.json",
        provider_states_setup_url="http://localhost:8000/pact/provider-states",
        enable_pending=False,
        publish_version="1.0.0",
        broker_url="http://localhost:9292",
        broker_username="pact",
        broker_password="pact"
    )

    assert success == 0, f"Verification failed: {logs}"

Provider State Setup

Provider needs to be in correct state for each test:

# app/api/pact_states.py

from fastapi import APIRouter

router = APIRouter(prefix="/pact")

@router.post("/provider-states")
async def setup_provider_state(request: dict):
    """Setup provider state for contract verification."""
    state = request.get("state")

    if state == "user exists with valid credentials":
        # Create test user in DB
        create_test_user("test@example.com", "testpass")

    elif state == "user does not exist":
        # Delete test user if exists
        delete_test_user("newuser@example.com")

    return {"result": "state setup complete"}

Run Provider Tests

# Ensure API is running
docker compose up -d voiceassist-server

# Run provider verification
pytest tests/contract/test_auth_contract.py::test_verify_pact_with_provider

Contract Testing Best Practices

1. Use Matchers, Not Exact Values

Bad:

body={"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."}  # Fragile

Good:

body={"access_token": Like("eyJ...")}  # Match type, not value

Available Matchers:

• Like(value): Match type and structure
• EachLike(value, minimum=1): Match array elements
• Term(regex, example): Match regex pattern
• Includes(value): Contains substring

2. Keep Contracts Minimal

Bad:

body={
    "id": "123",
    "email": "test@example.com",
    "full_name": "Test User",
    "created_at": "2025-11-21T...",
    "updated_at": "2025-11-21T...",
    "last_login": None,
    "preferences": {...},  # Too much detail
    "metadata": {...}
}

Good:

body={
    "id": Like("123"),
    "email": "test@example.com",
    "is_active": True
}  # Only what frontend needs

3. Use Provider States

Always define provider states for setup:

.given("user exists with email test@example.com")
.given("knowledge base has 10 documents")
.given("user is admin")

4. One Contract Per Interaction

Bad: Single test for login + get user profile Good: Separate tests for each API call

5. Test Both Success and Failure Cases

def test_login_success(): ...
def test_login_invalid_credentials(): ...
def test_login_inactive_user(): ...
def test_login_rate_limit_exceeded(): ...

CI/CD Integration

GitHub Actions Workflow

# .github/workflows/contract-tests.yml

name: Contract Tests

on: [push, pull_request]

jobs:
  consumer-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run consumer tests
        run: |
          pytest tests/contract/ -k Consumer
          pact-broker publish pacts/ \
            --consumer-app-version=$GITHUB_SHA \
            --broker-base-url=$PACT_BROKER_URL

  provider-tests:
    runs-on: ubuntu-latest
    needs: consumer-tests
    steps:
      - uses: actions/checkout@v3
      - name: Start services
        run: docker compose up -d
      - name: Run provider verification
        run: |
          pytest tests/contract/ -k Provider

Can-I-Deploy Check

Before deploying, verify contracts are met:

# Check if frontend v1.2.0 can work with API v2.0.0
pact-broker can-i-deploy \
  --pacticipant=VoiceAssistFrontend \
  --version=1.2.0 \
  --to-environment=production \
  --broker-base-url=http://localhost:9292 \
  --broker-username=pact \
  --broker-password=pact

Extending Contract Tests

Adding New Endpoint Contracts

1. Create Consumer Test:

# tests/contract/test_users_contract.py

def test_get_user_profile_contract(pact):
    (pact
     .given("user is authenticated")
     .upon_receiving("a request for user profile")
     .with_request(
         method="GET",
         path="/api/users/me",
         headers={"Authorization": Like("Bearer eyJ...")}
     )
     .will_respond_with(200, body={
         "id": Like("uuid"),
         "email": "user@example.com",
         "full_name": Like("John Doe")
     }))

    with pact:
        response = requests.get(f"{pact.uri}/api/users/me", ...)
        assert response.status_code == 200

2. Add Provider State:

# app/api/pact_states.py

@router.post("/provider-states")
async def setup_provider_state(request: dict):
    state = request.get("state")

    if state == "user is authenticated":
        # Create session, return token
        token = create_test_session()
        return {"token": token}

3. Run Tests:

pytest tests/contract/test_users_contract.py

Testing External Service Contracts

For Nextcloud, OpenAI, Qdrant:

# tests/contract/test_nextcloud_contract.py

def test_nextcloud_caldav_contract(pact):
    """Contract: Nextcloud CalDAV returns events."""
    (pact
     .given("calendar has 3 events")
     .upon_receiving("PROPFIND request for events")
     .with_request(
         method="PROPFIND",
         path="/remote.php/dav/calendars/user/personal/",
         headers={"Depth": "1"}
     )
     .will_respond_with(207, body=EachLike({
         "href": Like("/remote.php/dav/calendars/user/personal/event1.ics"),
         "etag": Like("\"12345\""),
         "calendar-data": Like("BEGIN:VCALENDAR...")
     }, minimum=1)))

    with pact:
        # Test VoiceAssist's Nextcloud client
        events = caldav_service.get_events()
        assert len(events) >= 1

Troubleshooting

Issue: Pact Broker Not Starting

Error: Connection refused to localhost:9292

Solutions:

# Check broker logs
docker compose logs pact-broker

# Verify PostgreSQL is running
docker compose ps postgres

# Create pact_broker database
docker compose exec postgres psql -U voiceassist -c "CREATE DATABASE pact_broker;"

# Restart broker
docker compose restart pact-broker

Issue: Provider Verification Fails

Error: Provider state setup failed

Solutions:

1. Implement provider state endpoint
2. Ensure API is running: docker compose up -d voiceassist-server
3. Check state setup logs: docker compose logs voiceassist-server

Issue: Contract Mismatch

Error: Expected status 200 but got 401

Root Causes:

• Consumer expects different response than provider returns
• Breaking API change not reflected in contract
• Provider state not set up correctly

Fix:

1. Update consumer test to match new API
2. Or update API to match contract
3. Publish new contract version

Contract Testing vs Integration Testing

Use Both: Contract tests for API stability, integration tests for workflows.

Related Documentation

• E2E Testing Guide - Full integration tests
• API Reference - OpenAPI specs
• Production Deployment Runbook - Production deployment

Document Version: 1.0 Last Updated: 2025-11-21 Maintained By: VoiceAssist QA Team Review Cycle: When API changes occur