Merged in feature/batch-status (pull request #215)
Implement and test the batch status feature * working
This commit is contained in:
Jay Brown
@@ -484,7 +484,7 @@ Lists batch uploads for a client with pagination support.
|
||||
|
||||
### `GET /client/{id}/document/batch/{batch_id}`
|
||||
|
||||
Retrieves detailed status information for a specific batch upload.
|
||||
Retrieves detailed status information for a specific batch upload, including per-file outcome tracking through the document processing pipeline.
|
||||
|
||||
**Security**: Requires JWT authentication
|
||||
|
||||
@@ -500,17 +500,89 @@ Retrieves detailed status information for a specific batch upload.
|
||||
"client_id": "AAA",
|
||||
"original_filename": "documents.zip",
|
||||
"status": "completed",
|
||||
"total_documents": 100,
|
||||
"processed_documents": 100,
|
||||
"failed_documents": 2,
|
||||
"total_documents": 5,
|
||||
"processed_documents": 3,
|
||||
"failed_documents": 1,
|
||||
"invalid_type_documents": 1,
|
||||
"progress_percent": 100,
|
||||
"created_at": "2024-01-01T00:00:00Z",
|
||||
"completed_at": "2024-01-01T00:05:00Z",
|
||||
"failed_filenames": ["doc3.pdf", "doc17.pdf"]
|
||||
"failed_filenames": ["corrupt.pdf"],
|
||||
"document_outcomes": [
|
||||
{
|
||||
"filename": "invoice.pdf",
|
||||
"outcome": "clean_passed",
|
||||
"document_id": "019580e0-1111-7676-8de9-000000000001",
|
||||
"created_at": "2024-01-01T00:00:01Z",
|
||||
"updated_at": "2024-01-01T00:04:50Z"
|
||||
},
|
||||
{
|
||||
"filename": "receipt.pdf",
|
||||
"outcome": "clean_passed",
|
||||
"document_id": "019580e0-2222-7676-8de9-000000000002",
|
||||
"created_at": "2024-01-01T00:00:01Z",
|
||||
"updated_at": "2024-01-01T00:04:52Z"
|
||||
},
|
||||
{
|
||||
"filename": "duplicate_invoice.pdf",
|
||||
"outcome": "duplicate",
|
||||
"document_id": "019580e0-1111-7676-8de9-000000000001",
|
||||
"created_at": "2024-01-01T00:00:01Z",
|
||||
"updated_at": "2024-01-01T00:00:01Z"
|
||||
},
|
||||
{
|
||||
"filename": "corrupt.pdf",
|
||||
"outcome": "clean_failed",
|
||||
"error_detail": "PDF validation failed",
|
||||
"document_id": "019580e0-3333-7676-8de9-000000000003",
|
||||
"clean_fail_reason": "invalid PDF structure: missing xref table",
|
||||
"created_at": "2024-01-01T00:00:01Z",
|
||||
"updated_at": "2024-01-01T00:04:55Z"
|
||||
},
|
||||
{
|
||||
"filename": "spreadsheet.xlsx",
|
||||
"outcome": "invalid_type",
|
||||
"error_detail": null,
|
||||
"document_id": null,
|
||||
"created_at": "2024-01-01T00:00:01Z",
|
||||
"updated_at": "2024-01-01T00:00:01Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `document_outcomes` array provides per-file tracking through the entire processing pipeline. Each file in the ZIP archive gets one outcome entry, starting at extraction time and updating as the file progresses through the document runners (init, sync, clean). When all outcomes reach a terminal state, the batch is fully processed.
|
||||
|
||||
**Document Outcome Values**:
|
||||
|
||||
| Outcome | Meaning | Terminal? |
|
||||
| ------------------ | ------------------------------------------------------ | --------- |
|
||||
| `submitted` | File extracted and uploaded, waiting for processing | No |
|
||||
| `duplicate` | File hash matches an existing document for this client | Yes |
|
||||
| `failed_open` | Could not open file from ZIP archive | Yes |
|
||||
| `failed_read` | Could not read file content from ZIP archive | Yes |
|
||||
| `failed_s3_upload` | Could not upload extracted file to S3 | Yes |
|
||||
| `failed_upload` | Upload handler returned an error | Yes |
|
||||
| `invalid_type` | File is not a PDF | Yes |
|
||||
| `init_complete` | Document record created, forwarded to sync | No |
|
||||
| `init_duplicate` | Duplicate detected during init processing | Yes |
|
||||
| `sync_complete` | Sync check passed, forwarded to clean | No |
|
||||
| `sync_skipped` | Client sync is disabled | Yes |
|
||||
| `clean_passed` | Document passed all validation checks | Yes |
|
||||
| `clean_failed` | Document failed validation (see `clean_fail_reason`) | Yes |
|
||||
|
||||
**Outcome Fields**:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------------- | -------------- | ------------------------------------------------------------------- |
|
||||
| `filename` | string | Original filename from the ZIP archive |
|
||||
| `outcome` | string | Current processing status (see table above) |
|
||||
| `error_detail` | string or null | Human-readable error message for failure outcomes |
|
||||
| `document_id` | uuid or null | Assigned document ID (null for files that never became documents) |
|
||||
| `clean_fail_reason` | string or null | Specific validation failure reason when `outcome` is `clean_failed` |
|
||||
| `created_at` | datetime | When the file was first extracted from the ZIP |
|
||||
| `updated_at` | datetime | When the outcome was last updated by a pipeline stage |
|
||||
|
||||
---
|
||||
|
||||
## Folder Operations (FolderService)
|
||||
|
||||
@@ -23,6 +23,7 @@ erDiagram
|
||||
|
||||
batch_uploads ||--o{ documents : "contains"
|
||||
batch_uploads ||--o{ documentUploads : "tracks uploads"
|
||||
batch_uploads ||--o{ batch_document_outcomes : "tracks outcomes"
|
||||
|
||||
folders ||--o{ documents : "contains"
|
||||
folders ||--o{ folders : "has children"
|
||||
@@ -457,6 +458,46 @@ CREATE INDEX idx_batch_uploads_status ON batch_uploads(status);
|
||||
- Indexed by client_id and status for efficient queries
|
||||
- Note: Uses snake_case naming convention (unlike other tables)
|
||||
|
||||
### Batch Document Outcomes (`batch_document_outcomes`)
|
||||
Tracks per-file extraction and pipeline processing outcomes for batch uploads. One row per file in the ZIP, updated in place as the document progresses through the processing pipeline (init, sync, clean runners).
|
||||
|
||||
```sql
|
||||
CREATE TYPE batch_document_outcome AS ENUM (
|
||||
'submitted', 'duplicate', 'failed_open', 'failed_read',
|
||||
'failed_s3_upload', 'failed_upload', 'invalid_type',
|
||||
'init_complete', 'init_duplicate', 'sync_complete',
|
||||
'sync_skipped', 'clean_passed', 'clean_failed'
|
||||
);
|
||||
|
||||
CREATE TABLE batch_document_outcomes (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||||
batch_id uuid NOT NULL,
|
||||
filename text NOT NULL,
|
||||
outcome batch_document_outcome NOT NULL,
|
||||
error_detail text,
|
||||
document_id uuid,
|
||||
updated_at timestamp NOT NULL DEFAULT NOW(),
|
||||
created_at timestamp NOT NULL DEFAULT NOW(),
|
||||
FOREIGN KEY (batch_id) REFERENCES batch_uploads(id),
|
||||
FOREIGN KEY (document_id) REFERENCES documents(id),
|
||||
UNIQUE (batch_id, filename)
|
||||
);
|
||||
|
||||
-- Indexes
|
||||
CREATE INDEX idx_batch_doc_outcomes_batch_id ON batch_document_outcomes(batch_id);
|
||||
CREATE INDEX idx_batch_doc_outcomes_document_id ON batch_document_outcomes(document_id);
|
||||
```
|
||||
|
||||
**Batch Document Outcome Features**:
|
||||
- Single source of truth for per-file batch status (extraction outcomes and pipeline progress)
|
||||
- ENUM type covers both extraction-time outcomes (submitted, duplicate, failed_*, invalid_type) and pipeline outcomes (init_*, sync_*, clean_*)
|
||||
- Unique constraint on (batch_id, filename) enables idempotent upserts for batch retries
|
||||
- Foreign key to documents table links outcome rows to their document records once created
|
||||
- Rows for files that never became documents (extraction failures) have null document_id
|
||||
- LEFT JOIN to `currentCleanEntries` provides specific clean failure reasons when queried
|
||||
- Indexed by batch_id for efficient batch detail lookups and by document_id for pipeline updates
|
||||
- Added in migration 124
|
||||
|
||||
### Folders (`folders`)
|
||||
Virtual folder hierarchy for document organization. Folders are database-only entities with no direct relationship to S3 storage locations.
|
||||
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
# Batch Outcome Tracking Guide
|
||||
|
||||
This guide walks through the batch document outcome tracking API, showing how to upload a batch of documents, monitor per-file processing progress, and interpret the results.
|
||||
|
||||
For API reference details, see [API Documentation - Batch Document Operations](./03-api-documentation.md#batch-document-operations-documentsservice).
|
||||
|
||||
## 1. Upload a Batch
|
||||
|
||||
Submit a ZIP archive containing PDF documents for batch processing.
|
||||
|
||||
```
|
||||
POST /client/AAA/document/batch
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: @mixed_documents.zip
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"batch_id": "019580df-ef65-7676-8de9-94435a93337a",
|
||||
"status": "processing",
|
||||
"status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
|
||||
}
|
||||
```
|
||||
|
||||
Use the `status_url` to poll for progress.
|
||||
|
||||
## 2. Poll for Extraction Results (Immediate)
|
||||
|
||||
```
|
||||
GET /client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a
|
||||
```
|
||||
|
||||
Response while the ZIP is being extracted:
|
||||
|
||||
```json
|
||||
{
|
||||
"batch_id": "019580df-ef65-7676-8de9-94435a93337a",
|
||||
"status": "processing",
|
||||
"progress_percent": 40,
|
||||
"total_documents": 5,
|
||||
"processed_documents": 0,
|
||||
"document_outcomes": [
|
||||
{ "filename": "invoice.pdf", "outcome": "submitted", "document_id": null },
|
||||
{ "filename": "receipt.pdf", "outcome": "submitted", "document_id": null }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
At this point, `submitted` means the file was extracted from the ZIP and uploaded to S3 but has not yet been processed by the document pipeline. Files that have not yet been extracted will not appear in `document_outcomes`.
|
||||
|
||||
## 3. Extraction Complete, Pipeline in Progress
|
||||
|
||||
Once ZIP extraction finishes, `status` changes to `completed` and all files appear in `document_outcomes`. Individual documents may still be progressing through the pipeline runners.
|
||||
|
||||
```json
|
||||
{
|
||||
"batch_id": "019580df-ef65-7676-8de9-94435a93337a",
|
||||
"status": "completed",
|
||||
"progress_percent": 100,
|
||||
"total_documents": 5,
|
||||
"processed_documents": 3,
|
||||
"failed_documents": 1,
|
||||
"invalid_type_documents": 1,
|
||||
"document_outcomes": [
|
||||
{
|
||||
"filename": "invoice.pdf",
|
||||
"outcome": "init_complete",
|
||||
"document_id": "...0001"
|
||||
},
|
||||
{
|
||||
"filename": "receipt.pdf",
|
||||
"outcome": "sync_complete",
|
||||
"document_id": "...0002"
|
||||
},
|
||||
{
|
||||
"filename": "duplicate_invoice.pdf",
|
||||
"outcome": "duplicate",
|
||||
"document_id": "...0001"
|
||||
},
|
||||
{
|
||||
"filename": "corrupt.pdf",
|
||||
"outcome": "init_complete",
|
||||
"document_id": "...0003"
|
||||
},
|
||||
{
|
||||
"filename": "spreadsheet.xlsx",
|
||||
"outcome": "invalid_type",
|
||||
"document_id": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Note: `status: "completed"` means the ZIP extraction is done. Individual documents may still be progressing through the pipeline. Check each file's `outcome` for its current stage.
|
||||
|
||||
## 4. Pipeline Fully Complete
|
||||
|
||||
When all documents have reached a terminal outcome, the batch is fully processed.
|
||||
|
||||
```json
|
||||
{
|
||||
"document_outcomes": [
|
||||
{
|
||||
"filename": "invoice.pdf",
|
||||
"outcome": "clean_passed",
|
||||
"document_id": "...0001"
|
||||
},
|
||||
{
|
||||
"filename": "receipt.pdf",
|
||||
"outcome": "clean_passed",
|
||||
"document_id": "...0002"
|
||||
},
|
||||
{
|
||||
"filename": "duplicate_invoice.pdf",
|
||||
"outcome": "duplicate",
|
||||
"document_id": "...0001"
|
||||
},
|
||||
{
|
||||
"filename": "corrupt.pdf",
|
||||
"outcome": "clean_failed",
|
||||
"document_id": "...0003",
|
||||
"error_detail": "PDF validation failed",
|
||||
"clean_fail_reason": "invalid PDF structure: missing xref table"
|
||||
},
|
||||
{
|
||||
"filename": "spreadsheet.xlsx",
|
||||
"outcome": "invalid_type",
|
||||
"document_id": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## 5. Interpreting Results for Resubmission
|
||||
|
||||
Each terminal outcome indicates a specific action for the caller:
|
||||
|
||||
- **`clean_passed`** -- Success. The document is fully processed. Use the `document_id` to retrieve it via `GET /document/{id}`.
|
||||
|
||||
- **`duplicate`** -- The file hash matches a document already in the system for this client. Use the returned `document_id` to reference the existing document. No resubmission needed.
|
||||
|
||||
- **`clean_failed`** -- The PDF was corrupt or failed validation. Check `clean_fail_reason` for the specific failure detail. Fix the source file and resubmit.
|
||||
|
||||
- **`invalid_type`** -- The file is not a PDF. Only PDF files are accepted.
|
||||
|
||||
- **`failed_open`** / **`failed_read`** -- The file inside the ZIP was damaged or unreadable. Re-create the ZIP archive and resubmit.
|
||||
|
||||
- **`failed_s3_upload`** / **`failed_upload`** -- An infrastructure error occurred during processing. Retry the entire batch.
|
||||
|
||||
- **`sync_skipped`** -- The client's sync configuration prevents processing. Contact an administrator.
|
||||
|
||||
- **`init_duplicate`** -- A duplicate was detected during the init processing stage (rather than at extraction time). The returned `document_id` references the existing document. No resubmission needed.
|
||||
|
||||
## 6. Outcome Lifecycle
|
||||
|
||||
Each file in a batch starts at extraction and progresses through the document processing pipeline. Terminal outcomes (marked below) are final states that will not change.
|
||||
|
||||
```
|
||||
ZIP extracted
|
||||
|
|
||||
+-- invalid_type (not PDF) .............. TERMINAL
|
||||
+-- failed_open / failed_read ........... TERMINAL
|
||||
+-- failed_s3_upload / failed_upload .... TERMINAL
|
||||
+-- duplicate (hash match) .............. TERMINAL
|
||||
+-- submitted (uploaded to S3)
|
||||
|
|
||||
+-- [docInitRunner]
|
||||
| +-- init_duplicate .......... TERMINAL
|
||||
| +-- init_complete
|
||||
| |
|
||||
| +-- [docSyncRunner]
|
||||
| +-- sync_skipped TERMINAL
|
||||
| +-- sync_complete
|
||||
| |
|
||||
| +-- [docCleanRunner]
|
||||
| +-- clean_failed .. TERMINAL
|
||||
| +-- clean_passed .. TERMINAL
|
||||
```
|
||||
|
||||
Non-terminal outcomes (`submitted`, `init_complete`, `sync_complete`) indicate the document is still being processed. Poll the batch details endpoint to see updates as files progress through each stage.
|
||||
@@ -186,7 +186,8 @@ This document provides a comprehensive summary of all REST API endpoints availab
|
||||
- Returns: `{batches[], total_count}`
|
||||
|
||||
- **GET /client/{id}/document/batch/{batch_id}** - Get detailed status of a specific batch upload
|
||||
- Returns: `{batch_id, client_id, original_filename, status, total_documents, processed_documents, failed_documents, invalid_type_documents, progress_percent, created_at, completed_at?, failed_filenames[]}`
|
||||
- Returns: `{batch_id, client_id, original_filename, status, total_documents, processed_documents, failed_documents, invalid_type_documents, progress_percent, created_at, completed_at?, failed_filenames[], document_outcomes[]}`
|
||||
- `document_outcomes` provides per-file tracking through the processing pipeline (see [Batch Outcome Tracking Guide](./batch-outcome-tracking-guide.md))
|
||||
|
||||
## Label Service
|
||||
|
||||
|
||||
Reference in New Issue
Block a user