query-orchestration/docs/ai.generated/02-service-architecture.md

# Service Architecture

## Service Catalog

### Primary Services

#### queryAPI - Main HTTP API Service
- **Purpose**: Central management API for all user entities and operations
- **Port**: 8080
- **Type**: HTTP REST API
- **Key Responsibilities**:
  - Client management and authentication
  - Document upload and metadata management (single and batch)
  - Batch upload processing with ZIP archive support
  - Query definition and testing
  - Export operations and status monitoring
  - Background task coordination and monitoring
  - User interface and API documentation

#### storeEventRunner - S3 Event Processor
- **Purpose**: Entry point for document processing pipeline
- **Port**: 8081
- **Type**: Event-driven queue consumer
- **Input**: S3 PUT event notifications (single files and batch uploads)
- **Output**: Events to DOCINIT queue
- **Responsibilities**: Processes S3 upload events, handles ZIP archive extraction, initiates document processing

#### docInitRunner - Document Initialization
- **Purpose**: Document registration and duplicate detection
- **Port**: 8082
- **Type**: Queue-based processor
- **Input**: `{bucket, key, hash}` from DOCINIT queue
- **Output**: Events to DOCSYNC queue
- **Key Functions**:
  - ETag-based duplicate detection
  - Document reference entity creation
  - Upload success logging

#### docSyncRunner - Document Sync Validation
- **Purpose**: Validates document eligibility for processing
- **Port**: 8083
- **Type**: Queue-based processor
- **Input**: `{document_id}` from DOCSYNC queue
- **Output**: Events to DOCCLEAN queue
- **Logic**: Validates client `can_sync` parameter

#### docCleanRunner - Document Preparation
- **Purpose**: Document cleaning and validation
- **Port**: 8084
- **Type**: Queue-based processor
- **Input**: `{document_id}` from DOCCLEAN queue
- **Output**: Events to DOCTEXT queue
- **Dependencies**: S3 ObjectStore for document access

#### docTextRunner - Text Extraction
- **Purpose**: OCR and text extraction using AWS Textract
- **Port**: 8085
- **Type**: Queue-based processor
- **Input**: `{document_id}` from DOCTEXT queue
- **Output**: Events to QUERYSYNC queue
- **Dependencies**: AWS Textract, S3 ObjectStore

#### querySyncRunner - Query Dependency Resolution
- **Purpose**: Identifies executable queries for documents
- **Port**: 8087
- **Type**: Queue-based processor
- **Input**: `{document_id}` from QUERYSYNC queue
- **Output**: Events to QUERY queue
- **Logic**: Finds queries with satisfied dependencies

#### queryRunner - Query Execution Engine
- **Purpose**: Executes queries and stores results
- **Port**: 8088
- **Type**: Queue-based processor
- **Input**: `{document_id, query_id}` from QUERY queue
- **Output**: Events to downstream QUERY queue
- **Responsibilities**: Query execution and result persistence

### Synchronization Services

#### clientSyncRunner - Client-Level Synchronization
- **Purpose**: Synchronizes all documents for a client
- **Port**: 8089
- **Type**: Queue-based processor
- **Input**: `{client_id}` from CLIENTSYNC queue
- **Output**: Events to DOCSYNC queue for each client document

#### queryVersionSyncRunner - Query Change Propagation
- **Purpose**: Propagates query updates to affected clients
- **Port**: 8090
- **Type**: Queue-based processor
- **Input**: `{query_id}` from QUERYVERSIONSYNC queue
- **Output**: Events to CLIENTSYNC queue for affected clients

### Utility Services

#### healthcheck - Health Monitoring
- **Purpose**: Service health validation
- **Type**: HTTP health check utility
- **Target**: localhost:8080 health endpoint

#### textExtractor - Development Utility
- **Purpose**: Local text extraction for testing
- **Type**: Standalone utility
- **Use Case**: Development and testing workflows

#### metricsExample_test - Monitoring Example
- **Purpose**: Demonstrates comprehensive Prometheus metrics implementation
- **Type**: Educational/testing service
- **Features**: Showcases 12 standard metric types (Counter, Gauge, Histogram, Summary)

#### Background Task Framework
- **Purpose**: Comprehensive background task execution system
- **Type**: Embedded framework within services
- **Features**: 
  - Periodic task execution with configurable intervals
  - On-demand task triggering via signals
  - Work coalescing to prevent duplicate processing
  - Statistics tracking and performance monitoring
  - Graceful shutdown with task completion

## Data Flow Architecture

### Document Processing Pipeline

[link to rendered version here](https://mermaid.live/edit#pako:eNp9k8FPgzAUxv-VhoMnPe3GYckGmCwxKLCpS7zU8rY1QottMVuW_e_2lWKGiBzIS78f3-v7Ss8BkyUEIQk0fLYgGMSc7hWt3wSxT0OV4Yw3VBgSVRyEGa8Xsz_WjFRAqCYai-TLfpi3QoAaoyvBDZKlZFhOYcVJMI9hOYVFFVDhOVdPgWs49m2xnMKyvq8NR53-65wh8ENOUfESkZga-k41dHr37uK9m8-LWUg2TSVpSWLJ2von9GKGKgYa2pq4VEkqDd9xRg2XwmNIWBLTDEn8GK3S1Rq31_p-KFg9XoYkOgD7IHHbVGgBegDgtM6g2KbRtQEK3uCZVry0X_rtO2kAuUNwLtFDskivbZzkB-4nJU9KMtCai_0QwkNyNuvkdTANCt4kh1FmXl28FKGrFWXjHh7KuoGzTZJvf4-cXc18z0VJkiOw1p5iBe7keZ9dD7rfwZsNjHDdG3UeQG78fclBt5XRwS0JalA15aW9mOfAHKB2V7SEHbVAcLl8AzPQPDE=)

```mermaid
sequenceDiagram
    participant Client
    participant S3
    participant Store as storeEventRunner
    participant Init as docInitRunner
    participant Sync as docSyncRunner
    participant Clean as docCleanRunner
    participant Text as docTextRunner
    participant QSync as querySyncRunner
    participant Query as queryRunner
    participant DB as Database
    
    Client->>S3: Upload Document
    S3->>Store: S3 Event Notification
    Store->>Init: DOCINIT Queue
    Init->>DB: Check Duplicates
    Init->>Sync: DOCSYNC Queue
    Sync->>DB: Validate Client Sync
    Sync->>Clean: DOCCLEAN Queue
    Clean->>S3: Document Processing
    Clean->>Text: DOCTEXT Queue
    Text->>S3: Read Document
    Text->>AWS: Textract Processing
    Text->>QSync: QUERYSYNC Queue
    QSync->>DB: Find Executable Queries
    QSync->>Query: QUERY Queue
    Query->>DB: Execute & Store Results
```

### Query Version Management Flow

[link to rendered version here](https://mermaid.live/edit#pako:eNp9kctqwzAQRX9l0KqF9ge8CBTZC0Nx_GgMgW6ENG4GLFmVZEII-ffKj_SVUi2EGJ0z3GHOTA4KWQLM4_uIRmJK4s0J_WogHitcIElWmABPSpP5o1zmIDxE2Z3i-xao2uYTaNF5GkxzMrIejUF3i_OZlj2hCf9xJVnsyeBEp4McNX4rLvhyz7kfN5sYLoGdVSIgVFMYSLEjQyEGWskyj1zMm0C1y-p9m9VNvi2afcEnY1zbRiBiPFL8Oc-Kl69_uLPoQHQdyoBqHeJ-sfgkXfMlkG75b2_BQa3DrN5V-WHXaN0g0Xs4UjhAgcdlJPYATKPTglTc6ZmFA-p5uwo7MfaBXS4faYamhw==)

```mermaid
sequenceDiagram
    participant Admin
    participant API as queryAPI
    participant QVS as queryVersionSyncRunner
    participant CS as clientSyncRunner
    participant Pipeline as Document Pipeline
    
    Admin->>API: Update Query Definition
    API->>QVS: QUERYVERSIONSYNC Queue
    QVS->>CS: CLIENTSYNC Queue (per affected client)
    CS->>Pipeline: DOCSYNC Queue (per client document)
    Pipeline->>Pipeline: Reprocess with New Query
```

## Communication Patterns

### Inter-Service Messaging
All services communicate through **AWS SQS queues** with the following patterns:
- **Fire-and-forget**: Services publish events without waiting for responses
- **Queue-based**: Loose coupling between services
- **Retry logic**: Built-in message retry and dead letter queue support
- **Ordering**: FIFO queues where order matters

### Queue Architecture

| Queue Name | Producer | Consumer | Message Format |
|------------|----------|----------|----------------|
| DOCINIT | storeEventRunner | docInitRunner | `{bucket, key, hash}` |
| DOCSYNC | docInitRunner, clientSyncRunner | docSyncRunner | `{document_id}` |
| DOCCLEAN | docSyncRunner | docCleanRunner | `{document_id}` |
| DOCTEXT | docCleanRunner | docTextRunner | `{document_id}` |
| QUERYSYNC | docTextRunner | querySyncRunner | `{document_id}` |
| QUERY | querySyncRunner, queryRunner | queryRunner | `{document_id, query_id}` |
| CLIENTSYNC | queryVersionSyncRunner | clientSyncRunner | `{client_id}` |
| QUERYVERSIONSYNC | queryAPI | queryVersionSyncRunner | `{query_id}` |
| BATCH_PROCESSING | queryAPI (batch worker) | Internal batch processor | `{batch_id, files[]}` |

### Database Access Patterns
- **Repository Pattern**: SQLC-generated type-safe database access
- **Connection Pooling**: Shared database connections across services
- **Transaction Management**: ACID transactions for data consistency
- **Read/Write Separation**: Potential for read replica scaling

## Scalability and Performance

### Horizontal Scaling Strategy
- **API Service**: Load balancer with multiple instances
- **Queue Processors**: Auto-scaling based on queue depth
- **Database**: Read replicas for query scaling
- **Storage**: S3 scales automatically

### Performance Optimizations
- **Connection Pooling**: Database connection reuse
- **Batch Processing**: Queue messages processed in batches
- **Caching**: JWKS caching for authentication
- **Compression**: Gzip compression for API responses

### Monitoring and Observability
- **Metrics**: Comprehensive Prometheus metrics with 12 standard types (Counter, Gauge, Histogram, Summary)
  - Request metrics, performance tracking, resource usage, dependency health
  - Cache operations, background tasks, queue metrics, error tracking
- **Tracing**: OpenTelemetry distributed tracing with correlation IDs
- **Logging**: Structured logging with comprehensive correlation
- **Health Checks**: Built-in health endpoints with dependency validation
- **Background Task Monitoring**: Task execution statistics and performance tracking