query-orchestration/docs/ai.generated/batch-outcome-tracking-guide.md

# Batch Outcome Tracking Guide

This guide walks through the batch document outcome tracking API, showing how to upload a batch of documents, monitor per-file processing progress, and interpret the results.

For API reference details, see [API Documentation - Batch Document Operations](./03-api-documentation.md#batch-document-operations-documentsservice).

## 1. Upload a Batch

Submit a ZIP archive containing documents for batch processing. Accepted file types: PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML.

```
POST /client/AAA/document/batch
Content-Type: multipart/form-data

file: @mixed_documents.zip
```

Response:

```json
{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
}
```

Use the `status_url` to poll for progress.

## 2. Poll for Extraction Results (Immediate)

```
GET /client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a
```

Response while the ZIP is being extracted:

```json
{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "progress_percent": 40,
  "total_documents": 5,
  "processed_documents": 0,
  "document_outcomes": [
    { "filename": "invoice.pdf", "outcome": "submitted", "document_id": null },
    { "filename": "receipt.pdf", "outcome": "submitted", "document_id": null }
  ]
}
```

At this point, `submitted` means the file was extracted from the ZIP and uploaded to S3 but has not yet been processed by the document pipeline. Files that have not yet been extracted will not appear in `document_outcomes`.

## 3. Extraction Complete, Pipeline in Progress

Once ZIP extraction finishes, `status` changes to `completed` and all files appear in `document_outcomes`. Individual documents may still be progressing through the pipeline runners.

```json
{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "completed",
  "progress_percent": 100,
  "total_documents": 5,
  "processed_documents": 3,
  "failed_documents": 1,
  "invalid_type_documents": 1,
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "init_complete",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "sync_complete",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "init_complete",
      "document_id": "...0003"
    },
    {
      "filename": "macro_report.xlsm",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}
```

Note: `status: "completed"` means the ZIP extraction is done. Individual documents may still be progressing through the pipeline. Check each file's `outcome` for its current stage.

## 4. Pipeline Fully Complete

When all documents have reached a terminal outcome, the batch is fully processed.

```json
{
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "clean_passed",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "clean_passed",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "clean_failed",
      "document_id": "...0003",
      "error_detail": "PDF validation failed",
      "clean_fail_reason": "invalid PDF structure: missing xref table"
    },
    {
      "filename": "macro_report.xlsm",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}
```

## 5. Interpreting Results for Resubmission

Each terminal outcome indicates a specific action for the caller:

- **`clean_passed`** -- Success. The document is fully processed. Use the `document_id` to retrieve it via `GET /document/{id}`.

- **`duplicate`** -- The file hash matches a document already in the system for this client. Use the returned `document_id` to reference the existing document. No resubmission needed.

- **`clean_failed`** -- The document was corrupt or failed validation. Check `clean_fail_reason` for the specific failure detail. Possible reasons include `invalid_mimetype`, `password_protected`, `contains_macros`, `empty_content`, `binary_content`, `unsupported_encoding`, `contains_attachments`, dimension/DPI issues, and more. Fix the source file and resubmit.

- **`invalid_type`** -- The file extension is not in the accepted list. Accepted extensions: `.pdf`, `.tiff`, `.tif`, `.jpeg`, `.jpg`, `.png`, `.bmp`, `.docx`, `.xlsx`, `.pptx`, `.txt`, `.eml`. Macro-enabled formats (`.docm`, `.xlsm`, `.pptm`) are not accepted.

- **`failed_open`** / **`failed_read`** -- The file inside the ZIP was damaged or unreadable. Re-create the ZIP archive and resubmit.

- **`failed_s3_upload`** / **`failed_upload`** -- An infrastructure error occurred during processing. `failed_upload` is also used for ZIP archives that contain no processable files after directories, hidden metadata, and unsafe paths are skipped.

- **`sync_skipped`** -- The client's sync configuration prevents processing. Contact an administrator.

- **`init_duplicate`** -- A duplicate was detected during the init processing stage (rather than at extraction time). The returned `document_id` references the existing document. No resubmission needed.

## 6. Outcome Lifecycle

Each file in a batch starts at extraction and progresses through the document processing pipeline. Terminal outcomes (marked below) are final states that will not change.

```
ZIP extracted
  |
  +-- invalid_type (unsupported ext) ....... TERMINAL
  +-- failed_open / failed_read ........... TERMINAL
  +-- failed_s3_upload / failed_upload .... TERMINAL
  +-- duplicate (hash match) .............. TERMINAL
  +-- submitted (uploaded to S3)
        |
        +-- [docInitRunner]
        |     +-- init_duplicate .......... TERMINAL
        |     +-- init_complete
        |           |
        |           +-- [docSyncRunner]
        |                 +-- sync_skipped  TERMINAL
        |                 +-- sync_complete
        |                       |
        |                       +-- [docCleanRunner]
        |                             +-- clean_failed .. TERMINAL
        |                             +-- clean_passed .. TERMINAL
```

Non-terminal outcomes (`submitted`, `init_complete`, `sync_complete`) indicate the document is still being processed. Poll the batch details endpoint to see updates as files progress through each stage.

## 7. Folder Cleanup Coupling

Legacy ZIP batch uploads still create folders during extraction. For batch uploads, the backend records touched folder segments in `batch_folder_candidates` and finalizes cleanup when the batch has terminal outcome evidence.

Accepted terminal outcomes (`clean_passed`, `sync_skipped`) keep their folder assignments. Rejected outcomes (`duplicate`, `init_duplicate`, `invalid_type`, `failed_open`, `failed_read`, `failed_s3_upload`, `failed_upload`, `clean_failed`) detach current-batch documents/uploads from auto-created candidate folders. Empty auto-created folders are then deleted only when they are not root, did not pre-exist, were not renamed, have no documents/uploads/children/chatbot scopes, and are not pending cleanup for another batch.

If outcome persistence fails before any outcome row exists, cleanup remains pending instead of marking `folder_cleanup_completed_at`. This keeps failed zero-outcome batches visible for retry/diagnosis and prevents cleanup from closing before rejected artifacts can be detached.