Jay Brown
6dccf494f8
cleanup permit policies and correct documentation * docs * policy cleanup * refactor * Merge remote-tracking branch 'origin/feature/permit-policy-cleanup' into feature/permit-policy-cleanup * docs and edits
117 lines
2.4 KiB
Markdown
117 lines
2.4 KiB
Markdown
# Service Processing Details
|
|
|
|
This document describes current service behavior in the processing pipeline.
|
|
|
|
## Active Pipeline Services
|
|
|
|
### 1. storeEventRunner
|
|
**Purpose**: Entry point for upload events.
|
|
|
|
**Code locations**:
|
|
- `cmd/storeEventRunner/`
|
|
- `api/storeEventRunner/`
|
|
- `internal/document/store/`
|
|
|
|
**Behavior**:
|
|
- Consumes S3 event notifications
|
|
- Validates supported event/object patterns
|
|
- Extracts bucket/key/hash metadata
|
|
- Publishes to `DOCINIT`
|
|
|
|
**Output**: `{bucket, key, hash}` to `DOCINIT`.
|
|
|
|
---
|
|
|
|
### 2. docInitRunner
|
|
**Purpose**: Document registration and deduplication.
|
|
|
|
**Code locations**:
|
|
- `cmd/docInitRunner/`
|
|
- `api/docInitRunner/`
|
|
- `internal/document/init/`
|
|
|
|
**Behavior**:
|
|
- Parses upload metadata from S3 key
|
|
- Deduplicates by client/hash
|
|
- Creates document and entry records for new documents
|
|
- Publishes new documents to `DOCSYNC`
|
|
|
|
**Output**: `{document_id}` to `DOCSYNC` for new documents.
|
|
|
|
---
|
|
|
|
### 3. docSyncRunner
|
|
**Purpose**: Sync eligibility gate.
|
|
|
|
**Code locations**:
|
|
- `cmd/docSyncRunner/`
|
|
- `api/docSyncRunner/`
|
|
- `internal/document/sync/`
|
|
|
|
**Behavior**:
|
|
- Loads document + client state
|
|
- Checks client sync status
|
|
- Publishes eligible documents to `DOCCLEAN`
|
|
|
|
**Output**: `{document_id}` to `DOCCLEAN` when eligible.
|
|
|
|
---
|
|
|
|
### 4. docCleanRunner
|
|
**Purpose**: Document validation/cleaning stage.
|
|
|
|
**Code locations**:
|
|
- `cmd/docCleanRunner/`
|
|
- `api/docCleanRunner/`
|
|
- `internal/document/clean/`
|
|
|
|
**Behavior**:
|
|
- Idempotent clean processing
|
|
- Validates MIME/content constraints and S3-backed inputs
|
|
- Stores clean version entries
|
|
|
|
**Output**: none. The active pipeline currently ends after clean.
|
|
|
|
---
|
|
|
|
### 5. clientSyncRunner
|
|
**Purpose**: Re-queue all documents for a client.
|
|
|
|
**Code locations**:
|
|
- `cmd/clientSyncRunner/`
|
|
- `api/clientSyncRunner/`
|
|
- `internal/client/sync/`
|
|
|
|
**Behavior**:
|
|
- Consumes `CLIENTSYNC`
|
|
- Enumerates client documents
|
|
- Publishes each to `DOCSYNC`
|
|
|
|
**Output**: `{document_id}` to `DOCSYNC` for each client document.
|
|
|
|
## Deprecated Compatibility Services
|
|
|
|
These services are retained for deployment compatibility and are currently health-only/no-op:
|
|
- `cmd/docTextRunner/`
|
|
- `cmd/querySyncRunner/`
|
|
- `cmd/queryRunner/`
|
|
- `cmd/queryVersionSyncRunner/`
|
|
|
|
## Active Data Flow Summary
|
|
|
|
```text
|
|
S3 Upload Event
|
|
-> storeEventRunner
|
|
-> docInitRunner
|
|
-> docSyncRunner
|
|
-> docCleanRunner
|
|
```
|
|
|
|
```text
|
|
Client Sync Trigger
|
|
-> clientSyncRunner
|
|
-> docSyncRunner
|
|
-> docCleanRunner
|
|
```
|
|
|