Jay Brown
6dccf494f8
cleanup permit policies and correct documentation * docs * policy cleanup * refactor * Merge remote-tracking branch 'origin/feature/permit-policy-cleanup' into feature/permit-policy-cleanup * docs and edits
112 lines
3.3 KiB
Markdown
112 lines
3.3 KiB
Markdown
# Service Architecture
|
|
|
|
## Service Catalog
|
|
|
|
### Active Services
|
|
|
|
#### queryAPI - Main HTTP API Service
|
|
- **Port**: 8080
|
|
- **Type**: HTTP REST API
|
|
- **Responsibilities**:
|
|
- Client, document, folder, label, collector, export, field extraction, admin, and EULA endpoints
|
|
- Authentication/authorization middleware integration (Cognito + Permit.io)
|
|
- Batch upload background processing coordination
|
|
|
|
#### storeEventRunner - S3 Event Processor
|
|
- **Port**: 8081
|
|
- **Type**: Queue consumer
|
|
- **Input**: S3 event notification messages
|
|
- **Output**: DOCINIT queue messages
|
|
|
|
#### docInitRunner - Document Initialization
|
|
- **Port**: 8082
|
|
- **Type**: Queue consumer
|
|
- **Input**: DOCINIT queue messages
|
|
- **Output**: DOCSYNC queue messages
|
|
- **Responsibilities**: document registration and deduplication by client/hash
|
|
|
|
#### docSyncRunner - Document Sync Gate
|
|
- **Port**: 8083
|
|
- **Type**: Queue consumer
|
|
- **Input**: DOCSYNC queue messages
|
|
- **Output**: DOCCLEAN queue messages
|
|
- **Responsibilities**: enforce client sync eligibility before processing
|
|
|
|
#### docCleanRunner - Document Validation/Cleaning
|
|
- **Port**: 8084
|
|
- **Type**: Queue consumer
|
|
- **Input**: DOCCLEAN queue messages
|
|
- **Output**: none (pipeline currently ends here)
|
|
- **Responsibilities**: content validation, clean record creation, idempotent processing
|
|
|
|
#### clientSyncRunner - Client Reprocessing Trigger
|
|
- **Port**: 8089
|
|
- **Type**: Queue consumer
|
|
- **Input**: CLIENTSYNC queue messages
|
|
- **Output**: DOCSYNC queue messages (one per client document)
|
|
|
|
### Deprecated Compatibility Services
|
|
|
|
The following services are retained as deployment compatibility placeholders and currently run health-only behavior:
|
|
- `docTextRunner` (8085)
|
|
- `querySyncRunner` (8087)
|
|
- `queryRunner` (8088)
|
|
- `queryVersionSyncRunner` (8090)
|
|
|
|
## Data Flow Architecture
|
|
|
|
### Active Document Processing Pipeline
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Client
|
|
participant S3
|
|
participant Store as storeEventRunner
|
|
participant Init as docInitRunner
|
|
participant Sync as docSyncRunner
|
|
participant Clean as docCleanRunner
|
|
|
|
Client->>S3: Upload Document
|
|
S3->>Store: S3 Event Notification
|
|
Store->>Init: DOCINIT Queue
|
|
Init->>Sync: DOCSYNC Queue
|
|
Sync->>Clean: DOCCLEAN Queue
|
|
```
|
|
|
|
### Client Synchronization Flow
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Admin
|
|
participant API as queryAPI
|
|
participant CS as clientSyncRunner
|
|
participant Sync as docSyncRunner
|
|
|
|
Admin->>API: Trigger client sync
|
|
API->>CS: CLIENTSYNC Queue
|
|
CS->>Sync: DOCSYNC Queue (per client document)
|
|
```
|
|
|
|
## Communication Patterns
|
|
|
|
### Inter-Service Messaging
|
|
- Services use **AWS SQS** for asynchronous decoupling
|
|
- Queue consumers are idempotent and retry-friendly
|
|
|
|
### Queue Architecture (Active)
|
|
|
|
| Queue Name | Producer | Consumer | Message Format |
|
|
|------------|----------|----------|----------------|
|
|
| DOCINIT | storeEventRunner | docInitRunner | `{bucket, key, hash}` |
|
|
| DOCSYNC | docInitRunner, clientSyncRunner | docSyncRunner | `{document_id}` |
|
|
| DOCCLEAN | docSyncRunner | docCleanRunner | `{document_id}` |
|
|
| CLIENTSYNC | queryAPI | clientSyncRunner | `{client_id}` |
|
|
|
|
### Queue Architecture (Deprecated Compatibility)
|
|
|
|
| Queue Name | Notes |
|
|
|------------|-------|
|
|
| DOCTEXT | Kept in some deployments for compatibility; not part of active pipeline |
|
|
| QUERYSYNC / QUERY / QUERYVERSIONSYNC | Legacy query pipeline queues; query subsystem removed |
|
|
|