query-orchestration/docs/ai.generated/batch-outcome-tracking-guide.md

Batch Outcome Tracking Guide

This guide walks through the batch document outcome tracking API, showing how to upload a batch of documents, monitor per-file processing progress, and interpret the results.

For API reference details, see API Documentation - Batch Document Operations.

1. Upload a Batch

Submit a ZIP archive containing PDF documents for batch processing.

POST /client/AAA/document/batch
Content-Type: multipart/form-data

file: @mixed_documents.zip

Response:

{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
}

Use the status_url to poll for progress.

2. Poll for Extraction Results (Immediate)

GET /client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a

Response while the ZIP is being extracted:

{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "progress_percent": 40,
  "total_documents": 5,
  "processed_documents": 0,
  "document_outcomes": [
    { "filename": "invoice.pdf", "outcome": "submitted", "document_id": null },
    { "filename": "receipt.pdf", "outcome": "submitted", "document_id": null }
  ]
}

At this point, submitted means the file was extracted from the ZIP and uploaded to S3 but has not yet been processed by the document pipeline. Files that have not yet been extracted will not appear in document_outcomes.

3. Extraction Complete, Pipeline in Progress

Once ZIP extraction finishes, status changes to completed and all files appear in document_outcomes. Individual documents may still be progressing through the pipeline runners.

{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "completed",
  "progress_percent": 100,
  "total_documents": 5,
  "processed_documents": 3,
  "failed_documents": 1,
  "invalid_type_documents": 1,
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "init_complete",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "sync_complete",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "init_complete",
      "document_id": "...0003"
    },
    {
      "filename": "spreadsheet.xlsx",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}

Note: status: "completed" means the ZIP extraction is done. Individual documents may still be progressing through the pipeline. Check each file's outcome for its current stage.

4. Pipeline Fully Complete

When all documents have reached a terminal outcome, the batch is fully processed.

{
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "clean_passed",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "clean_passed",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "clean_failed",
      "document_id": "...0003",
      "error_detail": "PDF validation failed",
      "clean_fail_reason": "invalid PDF structure: missing xref table"
    },
    {
      "filename": "spreadsheet.xlsx",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}

5. Interpreting Results for Resubmission

Each terminal outcome indicates a specific action for the caller:

• clean_passed -- Success. The document is fully processed. Use the document_id to retrieve it via GET /document/{id}.

• duplicate -- The file hash matches a document already in the system for this client. Use the returned document_id to reference the existing document. No resubmission needed.

• clean_failed -- The PDF was corrupt or failed validation. Check clean_fail_reason for the specific failure detail. Fix the source file and resubmit.

• invalid_type -- The file is not a PDF. Only PDF files are accepted.

• failed_open / failed_read -- The file inside the ZIP was damaged or unreadable. Re-create the ZIP archive and resubmit.

• failed_s3_upload / failed_upload -- An infrastructure error occurred during processing. Retry the entire batch.

• sync_skipped -- The client's sync configuration prevents processing. Contact an administrator.

• init_duplicate -- A duplicate was detected during the init processing stage (rather than at extraction time). The returned document_id references the existing document. No resubmission needed.

6. Outcome Lifecycle

Each file in a batch starts at extraction and progresses through the document processing pipeline. Terminal outcomes (marked below) are final states that will not change.

ZIP extracted
  |
  +-- invalid_type (not PDF) .............. TERMINAL
  +-- failed_open / failed_read ........... TERMINAL
  +-- failed_s3_upload / failed_upload .... TERMINAL
  +-- duplicate (hash match) .............. TERMINAL
  +-- submitted (uploaded to S3)
        |
        +-- [docInitRunner]
        |     +-- init_duplicate .......... TERMINAL
        |     +-- init_complete
        |           |
        |           +-- [docSyncRunner]
        |                 +-- sync_skipped  TERMINAL
        |                 +-- sync_complete
        |                       |
        |                       +-- [docCleanRunner]
        |                             +-- clean_failed .. TERMINAL
        |                             +-- clean_passed .. TERMINAL

Non-terminal outcomes (submitted, init_complete, sync_complete) indicate the document is still being processed. Poll the batch details endpoint to see updates as files progress through each stage.