Jacob Mathison
a080ca59d8
Implement v2 upload batch cleanup * Implement v2 upload batch cleanup * Merge remote-tracking branch 'origin/main' into jmathison/v2-upload-batch * Address upload batch review feedback * Raise batch worker coverage * Fix batch cleanup review issues Approved-by: Jay Brown
136 lines
3.2 KiB
Markdown
136 lines
3.2 KiB
Markdown
# Service Processing Details
|
|
|
|
This document describes current service behavior in the processing pipeline.
|
|
|
|
## Active Pipeline Services
|
|
|
|
### 1. storeEventRunner
|
|
**Purpose**: Entry point for upload events.
|
|
|
|
**Code locations**:
|
|
- `cmd/storeEventRunner/`
|
|
- `api/storeEventRunner/`
|
|
- `internal/document/store/`
|
|
|
|
**Behavior**:
|
|
- Consumes S3 event notifications
|
|
- Validates supported event/object patterns
|
|
- Extracts bucket/key/hash metadata
|
|
- Publishes to `DOCINIT`
|
|
|
|
**Output**: `{bucket, key, hash}` to `DOCINIT`.
|
|
|
|
---
|
|
|
|
### 2. docInitRunner
|
|
**Purpose**: Document registration and deduplication.
|
|
|
|
**Code locations**:
|
|
- `cmd/docInitRunner/`
|
|
- `api/docInitRunner/`
|
|
- `internal/document/init/`
|
|
|
|
**Behavior**:
|
|
- Parses upload metadata from S3 key
|
|
- Deduplicates by client/hash
|
|
- Creates document and entry records for new documents
|
|
- Publishes new documents to `DOCSYNC`
|
|
|
|
**Output**: `{document_id}` to `DOCSYNC` for new documents.
|
|
|
|
---
|
|
|
|
### 3. docSyncRunner
|
|
**Purpose**: Sync eligibility gate.
|
|
|
|
**Code locations**:
|
|
- `cmd/docSyncRunner/`
|
|
- `api/docSyncRunner/`
|
|
- `internal/document/sync/`
|
|
|
|
**Behavior**:
|
|
- Loads document + client state
|
|
- Checks client sync status
|
|
- Publishes eligible documents to `DOCCLEAN`
|
|
|
|
**Output**: `{document_id}` to `DOCCLEAN` when eligible.
|
|
|
|
---
|
|
|
|
### 4. docCleanRunner
|
|
**Purpose**: Document validation/cleaning stage.
|
|
|
|
**Code locations**:
|
|
- `cmd/docCleanRunner/`
|
|
- `api/docCleanRunner/`
|
|
- `internal/document/clean/`
|
|
|
|
**Behavior**:
|
|
- Idempotent clean processing
|
|
- Validates MIME/content constraints and S3-backed inputs
|
|
- Stores clean version entries
|
|
|
|
**Output**: none. The active pipeline currently ends after clean.
|
|
|
|
---
|
|
|
|
### 5. clientSyncRunner
|
|
**Purpose**: Re-queue all documents for a client.
|
|
|
|
**Code locations**:
|
|
- `cmd/clientSyncRunner/`
|
|
- `api/clientSyncRunner/`
|
|
- `internal/client/sync/`
|
|
|
|
**Behavior**:
|
|
- Consumes `CLIENTSYNC`
|
|
- Enumerates client documents
|
|
- Publishes each to `DOCSYNC`
|
|
|
|
**Output**: `{document_id}` to `DOCSYNC` for each client document.
|
|
|
|
---
|
|
|
|
### 6. batch folder cleanup
|
|
**Purpose**: Remove empty folders auto-created by legacy ZIP batch extraction when all file outcomes are terminal.
|
|
|
|
**Code locations**:
|
|
- `internal/document/batch/foldercleanup/`
|
|
- `internal/server/api/batch_worker.go`
|
|
- `internal/database/queries/batch.sql`
|
|
|
|
**Behavior**:
|
|
- Locks the batch row before finalizing cleanup
|
|
- Requires terminal batch status plus persisted outcome evidence
|
|
- Detaches rejected current-batch documents and upload rows from candidate folders
|
|
- Deletes only safe empty auto-created folders
|
|
- Retries ready cleanup from the QueryAPI background worker
|
|
|
|
**Output**: updates `batch_uploads.folder_cleanup_completed_at`; may null rejected `documents.folderId` and `documentUploads.folder_id`; may delete safe candidate folders.
|
|
|
|
## Deprecated Compatibility Services
|
|
|
|
These services are retained for deployment compatibility and are currently health-only/no-op:
|
|
- `cmd/docTextRunner/`
|
|
- `cmd/querySyncRunner/`
|
|
- `cmd/queryRunner/`
|
|
- `cmd/queryVersionSyncRunner/`
|
|
|
|
## Active Data Flow Summary
|
|
|
|
```text
|
|
S3 Upload Event
|
|
-> storeEventRunner
|
|
-> docInitRunner
|
|
-> docSyncRunner
|
|
-> docCleanRunner
|
|
-> batch folder cleanup (for legacy ZIP batches)
|
|
```
|
|
|
|
```text
|
|
Client Sync Trigger
|
|
-> clientSyncRunner
|
|
-> docSyncRunner
|
|
-> docCleanRunner
|
|
```
|