Jay Brown
da6de0080b
Update complete pdf code for entire system * docs and code for generation of updated docs
227 lines
9.2 KiB
Markdown
227 lines
9.2 KiB
Markdown
# Service Architecture
|
|
|
|
## Service Catalog
|
|
|
|
### Primary Services
|
|
|
|
#### queryAPI - Main HTTP API Service
|
|
- **Purpose**: Central management API for all user entities and operations
|
|
- **Port**: 8080
|
|
- **Type**: HTTP REST API
|
|
- **Key Responsibilities**:
|
|
- Client management and authentication
|
|
- Document upload and metadata management (single and batch)
|
|
- Batch upload processing with ZIP archive support
|
|
- Query definition and testing
|
|
- Export operations and status monitoring
|
|
- Background task coordination and monitoring
|
|
- User interface and API documentation
|
|
|
|
#### storeEventRunner - S3 Event Processor
|
|
- **Purpose**: Entry point for document processing pipeline
|
|
- **Port**: 8081
|
|
- **Type**: Event-driven queue consumer
|
|
- **Input**: S3 PUT event notifications (single files and batch uploads)
|
|
- **Output**: Events to DOCINIT queue
|
|
- **Responsibilities**: Processes S3 upload events, handles ZIP archive extraction, initiates document processing
|
|
|
|
#### docInitRunner - Document Initialization
|
|
- **Purpose**: Document registration and duplicate detection
|
|
- **Port**: 8082
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{bucket, key, hash}` from DOCINIT queue
|
|
- **Output**: Events to DOCSYNC queue
|
|
- **Key Functions**:
|
|
- ETag-based duplicate detection
|
|
- Document reference entity creation
|
|
- Upload success logging
|
|
|
|
#### docSyncRunner - Document Sync Validation
|
|
- **Purpose**: Validates document eligibility for processing
|
|
- **Port**: 8083
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{document_id}` from DOCSYNC queue
|
|
- **Output**: Events to DOCCLEAN queue
|
|
- **Logic**: Validates client `can_sync` parameter
|
|
|
|
#### docCleanRunner - Document Preparation
|
|
- **Purpose**: Document cleaning and validation
|
|
- **Port**: 8084
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{document_id}` from DOCCLEAN queue
|
|
- **Output**: Events to DOCTEXT queue
|
|
- **Dependencies**: S3 ObjectStore for document access
|
|
|
|
#### docTextRunner - Text Extraction
|
|
- **Purpose**: OCR and text extraction using AWS Textract
|
|
- **Port**: 8085
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{document_id}` from DOCTEXT queue
|
|
- **Output**: Events to QUERYSYNC queue
|
|
- **Dependencies**: AWS Textract, S3 ObjectStore
|
|
|
|
#### querySyncRunner - Query Dependency Resolution
|
|
- **Purpose**: Identifies executable queries for documents
|
|
- **Port**: 8087
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{document_id}` from QUERYSYNC queue
|
|
- **Output**: Events to QUERY queue
|
|
- **Logic**: Finds queries with satisfied dependencies
|
|
|
|
#### queryRunner - Query Execution Engine
|
|
- **Purpose**: Executes queries and stores results
|
|
- **Port**: 8088
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{document_id, query_id}` from QUERY queue
|
|
- **Output**: Events to downstream QUERY queue
|
|
- **Responsibilities**: Query execution and result persistence
|
|
|
|
### Synchronization Services
|
|
|
|
#### clientSyncRunner - Client-Level Synchronization
|
|
- **Purpose**: Synchronizes all documents for a client
|
|
- **Port**: 8089
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{client_id}` from CLIENTSYNC queue
|
|
- **Output**: Events to DOCSYNC queue for each client document
|
|
|
|
#### queryVersionSyncRunner - Query Change Propagation
|
|
- **Purpose**: Propagates query updates to affected clients
|
|
- **Port**: 8090
|
|
- **Type**: Queue-based processor
|
|
- **Input**: `{query_id}` from QUERYVERSIONSYNC queue
|
|
- **Output**: Events to CLIENTSYNC queue for affected clients
|
|
|
|
### Utility Services
|
|
|
|
#### healthcheck - Health Monitoring
|
|
- **Purpose**: Service health validation
|
|
- **Type**: HTTP health check utility
|
|
- **Target**: localhost:8080 health endpoint
|
|
|
|
#### textExtractor - Development Utility
|
|
- **Purpose**: Local text extraction for testing
|
|
- **Type**: Standalone utility
|
|
- **Use Case**: Development and testing workflows
|
|
|
|
#### metricsExample_test - Monitoring Example
|
|
- **Purpose**: Demonstrates comprehensive Prometheus metrics implementation
|
|
- **Type**: Educational/testing service
|
|
- **Features**: Showcases 12 standard metric types (Counter, Gauge, Histogram, Summary)
|
|
|
|
#### Background Task Framework
|
|
- **Purpose**: Comprehensive background task execution system
|
|
- **Type**: Embedded framework within services
|
|
- **Features**:
|
|
- Periodic task execution with configurable intervals
|
|
- On-demand task triggering via signals
|
|
- Work coalescing to prevent duplicate processing
|
|
- Statistics tracking and performance monitoring
|
|
- Graceful shutdown with task completion
|
|
|
|
## Data Flow Architecture
|
|
|
|
### Document Processing Pipeline
|
|
|
|
[link to rendered version here](https://mermaid.live/edit#pako:eNp9k8FPgzAUxv-VhoMnPe3GYckGmCwxKLCpS7zU8rY1QottMVuW_e_2lWKGiBzIS78f3-v7Ss8BkyUEIQk0fLYgGMSc7hWt3wSxT0OV4Yw3VBgSVRyEGa8Xsz_WjFRAqCYai-TLfpi3QoAaoyvBDZKlZFhOYcVJMI9hOYVFFVDhOVdPgWs49m2xnMKyvq8NR53-65wh8ENOUfESkZga-k41dHr37uK9m8-LWUg2TSVpSWLJ2von9GKGKgYa2pq4VEkqDd9xRg2XwmNIWBLTDEn8GK3S1Rq31_p-KFg9XoYkOgD7IHHbVGgBegDgtM6g2KbRtQEK3uCZVry0X_rtO2kAuUNwLtFDskivbZzkB-4nJU9KMtCai_0QwkNyNuvkdTANCt4kh1FmXl28FKGrFWXjHh7KuoGzTZJvf4-cXc18z0VJkiOw1p5iBe7keZ9dD7rfwZsNjHDdG3UeQG78fclBt5XRwS0JalA15aW9mOfAHKB2V7SEHbVAcLl8AzPQPDE=)
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Client
|
|
participant S3
|
|
participant Store as storeEventRunner
|
|
participant Init as docInitRunner
|
|
participant Sync as docSyncRunner
|
|
participant Clean as docCleanRunner
|
|
participant Text as docTextRunner
|
|
participant QSync as querySyncRunner
|
|
participant Query as queryRunner
|
|
participant DB as Database
|
|
|
|
Client->>S3: Upload Document
|
|
S3->>Store: S3 Event Notification
|
|
Store->>Init: DOCINIT Queue
|
|
Init->>DB: Check Duplicates
|
|
Init->>Sync: DOCSYNC Queue
|
|
Sync->>DB: Validate Client Sync
|
|
Sync->>Clean: DOCCLEAN Queue
|
|
Clean->>S3: Document Processing
|
|
Clean->>Text: DOCTEXT Queue
|
|
Text->>S3: Read Document
|
|
Text->>AWS: Textract Processing
|
|
Text->>QSync: QUERYSYNC Queue
|
|
QSync->>DB: Find Executable Queries
|
|
QSync->>Query: QUERY Queue
|
|
Query->>DB: Execute & Store Results
|
|
```
|
|
|
|
### Query Version Management Flow
|
|
|
|
[link to rendered version here](https://mermaid.live/edit#pako:eNp9kctqwzAQRX9l0KqF9ge8CBTZC0Nx_GgMgW6ENG4GLFmVZEII-ffKj_SVUi2EGJ0z3GHOTA4KWQLM4_uIRmJK4s0J_WogHitcIElWmABPSpP5o1zmIDxE2Z3i-xao2uYTaNF5GkxzMrIejUF3i_OZlj2hCf9xJVnsyeBEp4McNX4rLvhyz7kfN5sYLoGdVSIgVFMYSLEjQyEGWskyj1zMm0C1y-p9m9VNvi2afcEnY1zbRiBiPFL8Oc-Kl69_uLPoQHQdyoBqHeJ-sfgkXfMlkG75b2_BQa3DrN5V-WHXaN0g0Xs4UjhAgcdlJPYATKPTglTc6ZmFA-p5uwo7MfaBXS4faYamhw==)
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Admin
|
|
participant API as queryAPI
|
|
participant QVS as queryVersionSyncRunner
|
|
participant CS as clientSyncRunner
|
|
participant Pipeline as Document Pipeline
|
|
|
|
Admin->>API: Update Query Definition
|
|
API->>QVS: QUERYVERSIONSYNC Queue
|
|
QVS->>CS: CLIENTSYNC Queue (per affected client)
|
|
CS->>Pipeline: DOCSYNC Queue (per client document)
|
|
Pipeline->>Pipeline: Reprocess with New Query
|
|
```
|
|
|
|
## Communication Patterns
|
|
|
|
### Inter-Service Messaging
|
|
All services communicate through **AWS SQS queues** with the following patterns:
|
|
- **Fire-and-forget**: Services publish events without waiting for responses
|
|
- **Queue-based**: Loose coupling between services
|
|
- **Retry logic**: Built-in message retry and dead letter queue support
|
|
- **Ordering**: FIFO queues where order matters
|
|
|
|
### Queue Architecture
|
|
|
|
| Queue Name | Producer | Consumer | Message Format |
|
|
|------------|----------|----------|----------------|
|
|
| DOCINIT | storeEventRunner | docInitRunner | `{bucket, key, hash}` |
|
|
| DOCSYNC | docInitRunner, clientSyncRunner | docSyncRunner | `{document_id}` |
|
|
| DOCCLEAN | docSyncRunner | docCleanRunner | `{document_id}` |
|
|
| DOCTEXT | docCleanRunner | docTextRunner | `{document_id}` |
|
|
| QUERYSYNC | docTextRunner | querySyncRunner | `{document_id}` |
|
|
| QUERY | querySyncRunner, queryRunner | queryRunner | `{document_id, query_id}` |
|
|
| CLIENTSYNC | queryVersionSyncRunner | clientSyncRunner | `{client_id}` |
|
|
| QUERYVERSIONSYNC | queryAPI | queryVersionSyncRunner | `{query_id}` |
|
|
| BATCH_PROCESSING | queryAPI (batch worker) | Internal batch processor | `{batch_id, files[]}` |
|
|
|
|
### Database Access Patterns
|
|
- **Repository Pattern**: SQLC-generated type-safe database access
|
|
- **Connection Pooling**: Shared database connections across services
|
|
- **Transaction Management**: ACID transactions for data consistency
|
|
- **Read/Write Separation**: Potential for read replica scaling
|
|
|
|
## Scalability and Performance
|
|
|
|
### Horizontal Scaling Strategy
|
|
- **API Service**: Load balancer with multiple instances
|
|
- **Queue Processors**: Auto-scaling based on queue depth
|
|
- **Database**: Read replicas for query scaling
|
|
- **Storage**: S3 scales automatically
|
|
|
|
### Performance Optimizations
|
|
- **Connection Pooling**: Database connection reuse
|
|
- **Batch Processing**: Queue messages processed in batches
|
|
- **Caching**: JWKS caching for authentication
|
|
- **Compression**: Gzip compression for API responses
|
|
|
|
### Monitoring and Observability
|
|
- **Metrics**: Comprehensive Prometheus metrics with 12 standard types (Counter, Gauge, Histogram, Summary)
|
|
- Request metrics, performance tracking, resource usage, dependency health
|
|
- Cache operations, background tasks, queue metrics, error tracking
|
|
- **Tracing**: OpenTelemetry distributed tracing with correlation IDs
|
|
- **Logging**: Structured logging with comprehensive correlation
|
|
- **Health Checks**: Built-in health endpoints with dependency validation
|
|
- **Background Task Monitoring**: Task execution statistics and performance tracking |