query-orchestration/docs/ai.generated/02-service-architecture.md

Service Architecture

Service Catalog

Primary Services

queryAPI - Main HTTP API Service

• Purpose: Central management API for all user entities and operations
• Port: 8080
• Type: HTTP REST API
• Key Responsibilities:
  • Client management and authentication
  • Document upload and metadata management (single and batch)
  • Batch upload processing with ZIP archive support
  • Query definition and testing
  • Export operations and status monitoring
  • Background task coordination and monitoring
  • User interface and API documentation

storeEventRunner - S3 Event Processor

• Purpose: Entry point for document processing pipeline
• Port: 8081
• Type: Event-driven queue consumer
• Input: S3 PUT event notifications (single files and batch uploads)
• Output: Events to DOCINIT queue
• Responsibilities: Processes S3 upload events, handles ZIP archive extraction, initiates document processing

docInitRunner - Document Initialization

• Purpose: Document registration and duplicate detection
• Port: 8082
• Type: Queue-based processor
• Input: {bucket, key, hash} from DOCINIT queue
• Output: Events to DOCSYNC queue
• Key Functions:
  • ETag-based duplicate detection
  • Document reference entity creation
  • Upload success logging

docSyncRunner - Document Sync Validation

• Purpose: Validates document eligibility for processing
• Port: 8083
• Type: Queue-based processor
• Input: {document_id} from DOCSYNC queue
• Output: Events to DOCCLEAN queue
• Logic: Validates client can_sync parameter

docCleanRunner - Document Preparation

• Purpose: Document cleaning and validation
• Port: 8084
• Type: Queue-based processor
• Input: {document_id} from DOCCLEAN queue
• Output: Events to DOCTEXT queue
• Dependencies: S3 ObjectStore for document access

docTextRunner - Text Extraction

• Purpose: OCR and text extraction using AWS Textract
• Port: 8085
• Type: Queue-based processor
• Input: {document_id} from DOCTEXT queue
• Output: Events to QUERYSYNC queue
• Dependencies: AWS Textract, S3 ObjectStore

querySyncRunner - Query Dependency Resolution

• Purpose: Identifies executable queries for documents
• Port: 8087
• Type: Queue-based processor
• Input: {document_id} from QUERYSYNC queue
• Output: Events to QUERY queue
• Logic: Finds queries with satisfied dependencies

queryRunner - Query Execution Engine

• Purpose: Executes queries and stores results
• Port: 8088
• Type: Queue-based processor
• Input: {document_id, query_id} from QUERY queue
• Output: Events to downstream QUERY queue
• Responsibilities: Query execution and result persistence

Synchronization Services

clientSyncRunner - Client-Level Synchronization

• Purpose: Synchronizes all documents for a client
• Port: 8089
• Type: Queue-based processor
• Input: {client_id} from CLIENTSYNC queue
• Output: Events to DOCSYNC queue for each client document

queryVersionSyncRunner - Query Change Propagation

• Purpose: Propagates query updates to affected clients
• Port: 8090
• Type: Queue-based processor
• Input: {query_id} from QUERYVERSIONSYNC queue
• Output: Events to CLIENTSYNC queue for affected clients

Utility Services

healthcheck - Health Monitoring

• Purpose: Service health validation
• Type: HTTP health check utility
• Target: localhost:8080 health endpoint

textExtractor - Development Utility

• Purpose: Local text extraction for testing
• Type: Standalone utility
• Use Case: Development and testing workflows

metricsExample_test - Monitoring Example

• Purpose: Demonstrates comprehensive Prometheus metrics implementation
• Type: Educational/testing service
• Features: Showcases 12 standard metric types (Counter, Gauge, Histogram, Summary)

Background Task Framework

• Purpose: Comprehensive background task execution system
• Type: Embedded framework within services
• Features:
  • Periodic task execution with configurable intervals
  • On-demand task triggering via signals
  • Work coalescing to prevent duplicate processing
  • Statistics tracking and performance monitoring
  • Graceful shutdown with task completion

Data Flow Architecture

Document Processing Pipeline

link to rendered version here

sequenceDiagram
    participant Client
    participant S3
    participant Store as storeEventRunner
    participant Init as docInitRunner
    participant Sync as docSyncRunner
    participant Clean as docCleanRunner
    participant Text as docTextRunner
    participant QSync as querySyncRunner
    participant Query as queryRunner
    participant DB as Database
    
    Client->>S3: Upload Document
    S3->>Store: S3 Event Notification
    Store->>Init: DOCINIT Queue
    Init->>DB: Check Duplicates
    Init->>Sync: DOCSYNC Queue
    Sync->>DB: Validate Client Sync
    Sync->>Clean: DOCCLEAN Queue
    Clean->>S3: Document Processing
    Clean->>Text: DOCTEXT Queue
    Text->>S3: Read Document
    Text->>AWS: Textract Processing
    Text->>QSync: QUERYSYNC Queue
    QSync->>DB: Find Executable Queries
    QSync->>Query: QUERY Queue
    Query->>DB: Execute & Store Results

Query Version Management Flow

link to rendered version here

sequenceDiagram
    participant Admin
    participant API as queryAPI
    participant QVS as queryVersionSyncRunner
    participant CS as clientSyncRunner
    participant Pipeline as Document Pipeline
    
    Admin->>API: Update Query Definition
    API->>QVS: QUERYVERSIONSYNC Queue
    QVS->>CS: CLIENTSYNC Queue (per affected client)
    CS->>Pipeline: DOCSYNC Queue (per client document)
    Pipeline->>Pipeline: Reprocess with New Query

Communication Patterns

Inter-Service Messaging

All services communicate through AWS SQS queues with the following patterns:

• Fire-and-forget: Services publish events without waiting for responses
• Queue-based: Loose coupling between services
• Retry logic: Built-in message retry and dead letter queue support
• Ordering: FIFO queues where order matters

Queue Architecture

Queue Name	Producer	Consumer	Message Format
DOCINIT	storeEventRunner	docInitRunner	{bucket, key, hash}
DOCSYNC	docInitRunner, clientSyncRunner	docSyncRunner	{document_id}
DOCCLEAN	docSyncRunner	docCleanRunner	{document_id}
DOCTEXT	docCleanRunner	docTextRunner	{document_id}
QUERYSYNC	docTextRunner	querySyncRunner	{document_id}
QUERY	querySyncRunner, queryRunner	queryRunner	{document_id, query_id}
CLIENTSYNC	queryVersionSyncRunner	clientSyncRunner	{client_id}
QUERYVERSIONSYNC	queryAPI	queryVersionSyncRunner	{query_id}
BATCH_PROCESSING	queryAPI (batch worker)	Internal batch processor	{batch_id, files[]}

Database Access Patterns

• Repository Pattern: SQLC-generated type-safe database access
• Connection Pooling: Shared database connections across services
• Transaction Management: ACID transactions for data consistency
• Read/Write Separation: Potential for read replica scaling

Scalability and Performance

Horizontal Scaling Strategy

• API Service: Load balancer with multiple instances
• Queue Processors: Auto-scaling based on queue depth
• Database: Read replicas for query scaling
• Storage: S3 scales automatically

Performance Optimizations

• Connection Pooling: Database connection reuse
• Batch Processing: Queue messages processed in batches
• Caching: JWKS caching for authentication
• Compression: Gzip compression for API responses

Monitoring and Observability

• Metrics: Comprehensive Prometheus metrics with 12 standard types (Counter, Gauge, Histogram, Summary)
  • Request metrics, performance tracking, resource usage, dependency health
  • Cache operations, background tasks, queue metrics, error tracking
• Tracing: OpenTelemetry distributed tracing with correlation IDs
• Logging: Structured logging with comprehensive correlation
• Health Checks: Built-in health endpoints with dependency validation
• Background Task Monitoring: Task execution statistics and performance tracking