query-orchestration/docs/ai.generated/batch-outcome-tracking-guide.md

# Batch Outcome Tracking Guide

This guide walks through the batch document outcome tracking API, showing how to upload a batch of documents, monitor per-file processing progress, and interpret the results.

For API reference details, see [API Documentation - Batch Document Operations](./03-api-documentation.md#batch-document-operations-documentsservice).

## 1. Upload a Batch

Submit a ZIP archive containing PDF documents for batch processing.

```
POST /client/AAA/document/batch
Content-Type: multipart/form-data

file: @mixed_documents.zip
```

Response:

```json
{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
}
```

Use the `status_url` to poll for progress.

## 2. Poll for Extraction Results (Immediate)

```
GET /client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a
```

Response while the ZIP is being extracted:

```json
{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "progress_percent": 40,
  "total_documents": 5,
  "processed_documents": 0,
  "document_outcomes": [
    { "filename": "invoice.pdf", "outcome": "submitted", "document_id": null },
    { "filename": "receipt.pdf", "outcome": "submitted", "document_id": null }
  ]
}
```

At this point, `submitted` means the file was extracted from the ZIP and uploaded to S3 but has not yet been processed by the document pipeline. Files that have not yet been extracted will not appear in `document_outcomes`.

## 3. Extraction Complete, Pipeline in Progress

Once ZIP extraction finishes, `status` changes to `completed` and all files appear in `document_outcomes`. Individual documents may still be progressing through the pipeline runners.

```json
{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "completed",
  "progress_percent": 100,
  "total_documents": 5,
  "processed_documents": 3,
  "failed_documents": 1,
  "invalid_type_documents": 1,
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "init_complete",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "sync_complete",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "init_complete",
      "document_id": "...0003"
    },
    {
      "filename": "spreadsheet.xlsx",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}
```

Note: `status: "completed"` means the ZIP extraction is done. Individual documents may still be progressing through the pipeline. Check each file's `outcome` for its current stage.

## 4. Pipeline Fully Complete

When all documents have reached a terminal outcome, the batch is fully processed.

```json
{
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "clean_passed",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "clean_passed",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "clean_failed",
      "document_id": "...0003",
      "error_detail": "PDF validation failed",
      "clean_fail_reason": "invalid PDF structure: missing xref table"
    },
    {
      "filename": "spreadsheet.xlsx",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}
```

## 5. Interpreting Results for Resubmission

Each terminal outcome indicates a specific action for the caller:

- **`clean_passed`** -- Success. The document is fully processed. Use the `document_id` to retrieve it via `GET /document/{id}`.

- **`duplicate`** -- The file hash matches a document already in the system for this client. Use the returned `document_id` to reference the existing document. No resubmission needed.

- **`clean_failed`** -- The PDF was corrupt or failed validation. Check `clean_fail_reason` for the specific failure detail. Fix the source file and resubmit.

- **`invalid_type`** -- The file is not a PDF. Only PDF files are accepted.

- **`failed_open`** / **`failed_read`** -- The file inside the ZIP was damaged or unreadable. Re-create the ZIP archive and resubmit.

- **`failed_s3_upload`** / **`failed_upload`** -- An infrastructure error occurred during processing. Retry the entire batch.

- **`sync_skipped`** -- The client's sync configuration prevents processing. Contact an administrator.

- **`init_duplicate`** -- A duplicate was detected during the init processing stage (rather than at extraction time). The returned `document_id` references the existing document. No resubmission needed.

## 6. Outcome Lifecycle

Each file in a batch starts at extraction and progresses through the document processing pipeline. Terminal outcomes (marked below) are final states that will not change.

```
ZIP extracted
  |
  +-- invalid_type (not PDF) .............. TERMINAL
  +-- failed_open / failed_read ........... TERMINAL
  +-- failed_s3_upload / failed_upload .... TERMINAL
  +-- duplicate (hash match) .............. TERMINAL
  +-- submitted (uploaded to S3)
        |
        +-- [docInitRunner]
        |     +-- init_duplicate .......... TERMINAL
        |     +-- init_complete
        |           |
        |           +-- [docSyncRunner]
        |                 +-- sync_skipped  TERMINAL
        |                 +-- sync_complete
        |                       |
        |                       +-- [docCleanRunner]
        |                             +-- clean_failed .. TERMINAL
        |                             +-- clean_passed .. TERMINAL
```

Non-terminal outcomes (`submitted`, `init_complete`, `sync_complete`) indicate the document is still being processed. Poll the batch details endpoint to see updates as files progress through each stage.