Merged in feature/permit-policy-cleanup (pull request #210)
cleanup permit policies and correct documentation * docs * policy cleanup * refactor * Merge remote-tracking branch 'origin/feature/permit-policy-cleanup' into feature/permit-policy-cleanup * docs and edits
This commit is contained in:
Jay Brown
@@ -2,106 +2,60 @@
|
||||
|
||||
## Service Catalog
|
||||
|
||||
### Primary Services
|
||||
### Active Services
|
||||
|
||||
#### queryAPI - Main HTTP API Service
|
||||
- **Purpose**: Central management API for all user entities and operations
|
||||
- **Port**: 8080
|
||||
- **Type**: HTTP REST API
|
||||
- **Key Responsibilities**:
|
||||
- Client management and authentication
|
||||
- Document upload and metadata management (single and batch)
|
||||
- Batch upload processing with ZIP archive support
|
||||
- Folder and label management
|
||||
- Field extraction configuration
|
||||
- Export operations and status monitoring
|
||||
- Background task coordination and monitoring
|
||||
- User interface and API documentation
|
||||
- **Responsibilities**:
|
||||
- Client, document, folder, label, collector, export, field extraction, admin, and EULA endpoints
|
||||
- Authentication/authorization middleware integration (Cognito + Permit.io)
|
||||
- Batch upload background processing coordination
|
||||
|
||||
#### storeEventRunner - S3 Event Processor
|
||||
- **Purpose**: Entry point for document processing pipeline
|
||||
- **Port**: 8081
|
||||
- **Type**: Event-driven queue consumer
|
||||
- **Input**: S3 PUT event notifications (single files and batch uploads)
|
||||
- **Output**: Events to DOCINIT queue
|
||||
- **Responsibilities**: Processes S3 upload events, handles ZIP archive extraction, initiates document processing
|
||||
- **Type**: Queue consumer
|
||||
- **Input**: S3 event notification messages
|
||||
- **Output**: DOCINIT queue messages
|
||||
|
||||
#### docInitRunner - Document Initialization
|
||||
- **Purpose**: Document registration and duplicate detection
|
||||
- **Port**: 8082
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{bucket, key, hash}` from DOCINIT queue
|
||||
- **Output**: Events to DOCSYNC queue
|
||||
- **Key Functions**:
|
||||
- ETag-based duplicate detection
|
||||
- Document reference entity creation
|
||||
- Upload success logging
|
||||
- **Type**: Queue consumer
|
||||
- **Input**: DOCINIT queue messages
|
||||
- **Output**: DOCSYNC queue messages
|
||||
- **Responsibilities**: document registration and deduplication by client/hash
|
||||
|
||||
#### docSyncRunner - Document Sync Validation
|
||||
- **Purpose**: Validates document eligibility for processing
|
||||
#### docSyncRunner - Document Sync Gate
|
||||
- **Port**: 8083
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from DOCSYNC queue
|
||||
- **Output**: Events to DOCCLEAN queue
|
||||
- **Logic**: Validates client `can_sync` parameter
|
||||
- **Type**: Queue consumer
|
||||
- **Input**: DOCSYNC queue messages
|
||||
- **Output**: DOCCLEAN queue messages
|
||||
- **Responsibilities**: enforce client sync eligibility before processing
|
||||
|
||||
#### docCleanRunner - Document Preparation
|
||||
- **Purpose**: Document cleaning and validation
|
||||
#### docCleanRunner - Document Validation/Cleaning
|
||||
- **Port**: 8084
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from DOCCLEAN queue
|
||||
- **Output**: Events to DOCTEXT queue
|
||||
- **Dependencies**: S3 ObjectStore for document access
|
||||
- **Type**: Queue consumer
|
||||
- **Input**: DOCCLEAN queue messages
|
||||
- **Output**: none (pipeline currently ends here)
|
||||
- **Responsibilities**: content validation, clean record creation, idempotent processing
|
||||
|
||||
#### docTextRunner - Text Extraction
|
||||
- **Purpose**: OCR and text extraction using AWS Textract
|
||||
- **Port**: 8085
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from DOCTEXT queue
|
||||
- **Output**: Text stored in database, processing complete
|
||||
- **Dependencies**: AWS Textract, S3 ObjectStore
|
||||
|
||||
### Synchronization Services
|
||||
|
||||
#### clientSyncRunner - Client-Level Synchronization
|
||||
- **Purpose**: Synchronizes all documents for a client
|
||||
#### clientSyncRunner - Client Reprocessing Trigger
|
||||
- **Port**: 8089
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{client_id}` from CLIENTSYNC queue
|
||||
- **Output**: Events to DOCSYNC queue for each client document
|
||||
- **Type**: Queue consumer
|
||||
- **Input**: CLIENTSYNC queue messages
|
||||
- **Output**: DOCSYNC queue messages (one per client document)
|
||||
|
||||
### Utility Services
|
||||
### Deprecated Compatibility Services
|
||||
|
||||
#### healthcheck - Health Monitoring
|
||||
- **Purpose**: Service health validation
|
||||
- **Type**: HTTP health check utility
|
||||
- **Target**: localhost:8080 health endpoint
|
||||
|
||||
#### textExtractor - Development Utility
|
||||
- **Purpose**: Local text extraction for testing
|
||||
- **Type**: Standalone utility
|
||||
- **Use Case**: Development and testing workflows
|
||||
|
||||
#### metricsExample_test - Monitoring Example
|
||||
- **Purpose**: Demonstrates comprehensive Prometheus metrics implementation
|
||||
- **Type**: Educational/testing service
|
||||
- **Features**: Showcases 12 standard metric types (Counter, Gauge, Histogram, Summary)
|
||||
|
||||
#### Background Task Framework
|
||||
- **Purpose**: Comprehensive background task execution system
|
||||
- **Type**: Embedded framework within services
|
||||
- **Features**:
|
||||
- Periodic task execution with configurable intervals
|
||||
- On-demand task triggering via signals
|
||||
- Work coalescing to prevent duplicate processing
|
||||
- Statistics tracking and performance monitoring
|
||||
- Graceful shutdown with task completion
|
||||
The following services are retained as deployment compatibility placeholders and currently run health-only behavior:
|
||||
- `docTextRunner` (8085)
|
||||
- `querySyncRunner` (8087)
|
||||
- `queryRunner` (8088)
|
||||
- `queryVersionSyncRunner` (8090)
|
||||
|
||||
## Data Flow Architecture
|
||||
|
||||
### Document Processing Pipeline
|
||||
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNp9k8FPgzAUxv-VhoMnPe3GYckGmCwxKLCpS7zU8rY1QottMVuW_e_2lWKGiBzIS78f3-v7Ss8BkyUEIQk0fLYgGMSc7hWt3wSxT0OV4Yw3VBgSVRyEGa8Xsz_WjFRAqCYai-TLfpi3QoAaoyvBDZKlZFhOYcVJMI9hOYVFFVDhOVdPgWs49m2xnMKyvq8NR53-65wh8ENOUfESkZga-k41dHr37uK9m8-LWUg2TSVpSWLJ2von9GKGKgYa2pq4VEkqDd9xRg2XwmNIWBLTDEn8GK3S1Rq31_p-KFg9XoYkOgD7IHHbVGgBegDgtM6g2KbRtQEK3uCZVry0X_rtO2kAuUNwLtFDskivbZzkB-4nJU9KMtCai_0QwkNyNuvkdTANCt4kh1FmXl28FKGrFWXjHh7KuoGzTZJvf4-cXc18z0VJkiOw1p5iBe7keZ9dD7rfwZsNjHDdG3UeQG78fclBt5XRwS0JalA15aW9mOfAHKB2V7SEHbVAcLl8AzPQPDE=)
|
||||
### Active Document Processing Pipeline
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
@@ -111,21 +65,12 @@ sequenceDiagram
|
||||
participant Init as docInitRunner
|
||||
participant Sync as docSyncRunner
|
||||
participant Clean as docCleanRunner
|
||||
participant Text as docTextRunner
|
||||
participant DB as Database
|
||||
|
||||
Client->>S3: Upload Document
|
||||
S3->>Store: S3 Event Notification
|
||||
Store->>Init: DOCINIT Queue
|
||||
Init->>DB: Check Duplicates
|
||||
Init->>Sync: DOCSYNC Queue
|
||||
Sync->>DB: Validate Client Sync
|
||||
Sync->>Clean: DOCCLEAN Queue
|
||||
Clean->>S3: Document Processing
|
||||
Clean->>Text: DOCTEXT Queue
|
||||
Text->>S3: Read Document
|
||||
Text->>AWS: Textract Processing
|
||||
Text->>DB: Store Extracted Text
|
||||
```
|
||||
|
||||
### Client Synchronization Flow
|
||||
@@ -135,59 +80,32 @@ sequenceDiagram
|
||||
participant Admin
|
||||
participant API as queryAPI
|
||||
participant CS as clientSyncRunner
|
||||
participant Pipeline as Document Pipeline
|
||||
participant Sync as docSyncRunner
|
||||
|
||||
Admin->>API: Update Client Configuration
|
||||
Admin->>API: Trigger client sync
|
||||
API->>CS: CLIENTSYNC Queue
|
||||
CS->>Pipeline: DOCSYNC Queue (per client document)
|
||||
Pipeline->>Pipeline: Reprocess Documents
|
||||
CS->>Sync: DOCSYNC Queue (per client document)
|
||||
```
|
||||
|
||||
## Communication Patterns
|
||||
|
||||
### Inter-Service Messaging
|
||||
All services communicate through **AWS SQS queues** with the following patterns:
|
||||
- **Fire-and-forget**: Services publish events without waiting for responses
|
||||
- **Queue-based**: Loose coupling between services
|
||||
- **Retry logic**: Built-in message retry and dead letter queue support
|
||||
- **Ordering**: FIFO queues where order matters
|
||||
- Services use **AWS SQS** for asynchronous decoupling
|
||||
- Queue consumers are idempotent and retry-friendly
|
||||
|
||||
### Queue Architecture
|
||||
### Queue Architecture (Active)
|
||||
|
||||
| Queue Name | Producer | Consumer | Message Format |
|
||||
|------------|----------|----------|----------------|
|
||||
| DOCINIT | storeEventRunner | docInitRunner | `{bucket, key, hash}` |
|
||||
| DOCSYNC | docInitRunner, clientSyncRunner | docSyncRunner | `{document_id}` |
|
||||
| DOCCLEAN | docSyncRunner | docCleanRunner | `{document_id}` |
|
||||
| DOCTEXT | docCleanRunner | docTextRunner | `{document_id}` |
|
||||
| CLIENTSYNC | queryAPI | clientSyncRunner | `{client_id}` |
|
||||
| BATCH_PROCESSING | queryAPI (batch worker) | Internal batch processor | `{batch_id, files[]}` |
|
||||
|
||||
### Database Access Patterns
|
||||
- **Repository Pattern**: SQLC-generated type-safe database access
|
||||
- **Connection Pooling**: Shared database connections across services
|
||||
- **Transaction Management**: ACID transactions for data consistency
|
||||
- **Read/Write Separation**: Potential for read replica scaling
|
||||
### Queue Architecture (Deprecated Compatibility)
|
||||
|
||||
## Scalability and Performance
|
||||
| Queue Name | Notes |
|
||||
|------------|-------|
|
||||
| DOCTEXT | Kept in some deployments for compatibility; not part of active pipeline |
|
||||
| QUERYSYNC / QUERY / QUERYVERSIONSYNC | Legacy query pipeline queues; query subsystem removed |
|
||||
|
||||
### Horizontal Scaling Strategy
|
||||
- **API Service**: Load balancer with multiple instances
|
||||
- **Queue Processors**: Auto-scaling based on queue depth
|
||||
- **Database**: Read replicas for scaling
|
||||
- **Storage**: S3 scales automatically
|
||||
|
||||
### Performance Optimizations
|
||||
- **Connection Pooling**: Database connection reuse
|
||||
- **Batch Processing**: Queue messages processed in batches
|
||||
- **Caching**: JWKS caching for authentication
|
||||
- **Compression**: Gzip compression for API responses
|
||||
|
||||
### Monitoring and Observability
|
||||
- **Metrics**: Comprehensive Prometheus metrics with 12 standard types (Counter, Gauge, Histogram, Summary)
|
||||
- Request metrics, performance tracking, resource usage, dependency health
|
||||
- Cache operations, background tasks, queue metrics, error tracking
|
||||
- **Tracing**: OpenTelemetry distributed tracing with correlation IDs
|
||||
- **Logging**: Structured logging with comprehensive correlation
|
||||
- **Health Checks**: Built-in health endpoints with dependency validation
|
||||
- **Background Task Monitoring**: Task execution statistics and performance tracking
|
||||
Reference in New Issue
Block a user