jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in feature/batch-status (pull request #215)

Browse Source 
Implement and test the batch status feature

* working
This commit is contained in:
Jay Brown
2026-03-12 18:53:42 +00:00
parent 8a4029058b
commit aafe7d5b5f
39 changed files with 4252 additions and 516 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ai.generated/03-api-documentation.md
+77 -5
View File
@@ -484,7 +484,7 @@ Lists batch uploads for a client with pagination support.
### `GET /client/{id}/document/batch/{batch_id}`
Retrieves detailed status information for a specific batch upload.
Retrieves detailed status information for a specific batch upload, including per-file outcome tracking through the document processing pipeline.
**Security**: Requires JWT authentication
@@ -500,17 +500,89 @@ Retrieves detailed status information for a specific batch upload.
"client_id": "AAA",
"original_filename": "documents.zip",
"status": "completed",
"total_documents": 100,
"processed_documents": 100,
"failed_documents": 2,
"total_documents": 5,
"processed_documents": 3,
"failed_documents": 1,
"invalid_type_documents": 1,
"progress_percent": 100,
"created_at": "2024-01-01T00:00:00Z",
"completed_at": "2024-01-01T00:05:00Z",
"failed_filenames": ["doc3.pdf", "doc17.pdf"]
"failed_filenames": ["corrupt.pdf"],
"document_outcomes": [
{
"filename": "invoice.pdf",
"outcome": "clean_passed",
"document_id": "019580e0-1111-7676-8de9-000000000001",
"created_at": "2024-01-01T00:00:01Z",
"updated_at": "2024-01-01T00:04:50Z"
},
{
"filename": "receipt.pdf",
"outcome": "clean_passed",
"document_id": "019580e0-2222-7676-8de9-000000000002",
"created_at": "2024-01-01T00:00:01Z",
"updated_at": "2024-01-01T00:04:52Z"
},
{
"filename": "duplicate_invoice.pdf",
"outcome": "duplicate",
"document_id": "019580e0-1111-7676-8de9-000000000001",
"created_at": "2024-01-01T00:00:01Z",
"updated_at": "2024-01-01T00:00:01Z"
},
{
"filename": "corrupt.pdf",
"outcome": "clean_failed",
"error_detail": "PDF validation failed",
"document_id": "019580e0-3333-7676-8de9-000000000003",
"clean_fail_reason": "invalid PDF structure: missing xref table",
"created_at": "2024-01-01T00:00:01Z",
"updated_at": "2024-01-01T00:04:55Z"
},
{
"filename": "spreadsheet.xlsx",
"outcome": "invalid_type",
"error_detail": null,
"document_id": null,
"created_at": "2024-01-01T00:00:01Z",
"updated_at": "2024-01-01T00:00:01Z"
}
]
}
```
The `document_outcomes` array provides per-file tracking through the entire processing pipeline. Each file in the ZIP archive gets one outcome entry, starting at extraction time and updating as the file progresses through the document runners (init, sync, clean). When all outcomes reach a terminal state, the batch is fully processed.
**Document Outcome Values**:
| Outcome | Meaning | Terminal? |
| ------------------ | ------------------------------------------------------ | --------- |
| `submitted` | File extracted and uploaded, waiting for processing | No |
| `duplicate` | File hash matches an existing document for this client | Yes |
| `failed_open` | Could not open file from ZIP archive | Yes |
| `failed_read` | Could not read file content from ZIP archive | Yes |
| `failed_s3_upload` | Could not upload extracted file to S3 | Yes |
| `failed_upload` | Upload handler returned an error | Yes |
| `invalid_type` | File is not a PDF | Yes |
| `init_complete` | Document record created, forwarded to sync | No |
| `init_duplicate` | Duplicate detected during init processing | Yes |
| `sync_complete` | Sync check passed, forwarded to clean | No |
| `sync_skipped` | Client sync is disabled | Yes |
| `clean_passed` | Document passed all validation checks | Yes |
| `clean_failed` | Document failed validation (see `clean_fail_reason`) | Yes |
**Outcome Fields**:
| Field | Type | Description |
| ------------------- | -------------- | ------------------------------------------------------------------- |
| `filename` | string | Original filename from the ZIP archive |
| `outcome` | string | Current processing status (see table above) |
| `error_detail` | string or null | Human-readable error message for failure outcomes |
| `document_id` | uuid or null | Assigned document ID (null for files that never became documents) |
| `clean_fail_reason` | string or null | Specific validation failure reason when `outcome` is `clean_failed` |
| `created_at` | datetime | When the file was first extracted from the ZIP |
| `updated_at` | datetime | When the outcome was last updated by a pipeline stage |
---
## Folder Operations (FolderService)