query-orchestration/docs/ai.generated/batch-outcome-tracking-guide.md

Batch Outcome Tracking Guide

This guide walks through the batch document outcome tracking API, showing how to upload a batch of documents, monitor per-file processing progress, and interpret the results.

For API reference details, see API Documentation - Batch Document Operations.

1. Upload a Batch

Submit a ZIP archive containing documents for batch processing. Accepted file types: PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML.

POST /client/AAA/document/batch
Content-Type: multipart/form-data

file: @mixed_documents.zip

Response:

{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
}

Use the status_url to poll for progress.

2. Poll for Extraction Results (Immediate)

GET /client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a

Response while the ZIP is being extracted:

{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "processing",
  "progress_percent": 40,
  "total_documents": 5,
  "processed_documents": 0,
  "document_outcomes": [
    { "filename": "invoice.pdf", "outcome": "submitted", "document_id": null },
    { "filename": "receipt.pdf", "outcome": "submitted", "document_id": null }
  ]
}

At this point, submitted means the file was extracted from the ZIP and uploaded to S3 but has not yet been processed by the document pipeline. Files that have not yet been extracted will not appear in document_outcomes.

3. Extraction Complete, Pipeline in Progress

Once ZIP extraction finishes, status changes to completed and all files appear in document_outcomes. Individual documents may still be progressing through the pipeline runners.

{
  "batch_id": "019580df-ef65-7676-8de9-94435a93337a",
  "status": "completed",
  "progress_percent": 100,
  "total_documents": 5,
  "processed_documents": 3,
  "failed_documents": 1,
  "invalid_type_documents": 1,
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "init_complete",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "sync_complete",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "init_complete",
      "document_id": "...0003"
    },
    {
      "filename": "macro_report.xlsm",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}

Note: status: "completed" means the ZIP extraction is done. Individual documents may still be progressing through the pipeline. Check each file's outcome for its current stage.

4. Pipeline Fully Complete

When all documents have reached a terminal outcome, the batch is fully processed.

{
  "document_outcomes": [
    {
      "filename": "invoice.pdf",
      "outcome": "clean_passed",
      "document_id": "...0001"
    },
    {
      "filename": "receipt.pdf",
      "outcome": "clean_passed",
      "document_id": "...0002"
    },
    {
      "filename": "duplicate_invoice.pdf",
      "outcome": "duplicate",
      "document_id": "...0001"
    },
    {
      "filename": "corrupt.pdf",
      "outcome": "clean_failed",
      "document_id": "...0003",
      "error_detail": "PDF validation failed",
      "clean_fail_reason": "invalid PDF structure: missing xref table"
    },
    {
      "filename": "macro_report.xlsm",
      "outcome": "invalid_type",
      "document_id": null
    }
  ]
}

5. Interpreting Results for Resubmission

Each terminal outcome indicates a specific action for the caller:

• clean_passed -- Success. The document is fully processed. Use the document_id to retrieve it via GET /document/{id}.

• duplicate -- The file hash matches a document already in the system for this client. Use the returned document_id to reference the existing document. No resubmission needed.

• clean_failed -- The document was corrupt or failed validation. Check clean_fail_reason for the specific failure detail. Possible reasons include invalid_mimetype, password_protected, contains_macros, empty_content, binary_content, unsupported_encoding, contains_attachments, dimension/DPI issues, and more. Fix the source file and resubmit.

• invalid_type -- The file extension is not in the accepted list. Accepted extensions: .pdf, .tiff, .tif, .jpeg, .jpg, .png, .bmp, .docx, .xlsx, .pptx, .txt, .eml. Macro-enabled formats (.docm, .xlsm, .pptm) are not accepted.

• failed_open / failed_read -- The file inside the ZIP was damaged or unreadable. Re-create the ZIP archive and resubmit.

• failed_s3_upload / failed_upload -- An infrastructure error occurred during processing. failed_upload is also used for ZIP archives that contain no processable files after directories, hidden metadata, and unsafe paths are skipped.

• sync_skipped -- The client's sync configuration prevents processing. Contact an administrator.

• init_duplicate -- A duplicate was detected during the init processing stage (rather than at extraction time). The returned document_id references the existing document. No resubmission needed.

6. Outcome Lifecycle

Each file in a batch starts at extraction and progresses through the document processing pipeline. Terminal outcomes (marked below) are final states that will not change.

ZIP extracted
  |
  +-- invalid_type (unsupported ext) ....... TERMINAL
  +-- failed_open / failed_read ........... TERMINAL
  +-- failed_s3_upload / failed_upload .... TERMINAL
  +-- duplicate (hash match) .............. TERMINAL
  +-- submitted (uploaded to S3)
        |
        +-- [docInitRunner]
        |     +-- init_duplicate .......... TERMINAL
        |     +-- init_complete
        |           |
        |           +-- [docSyncRunner]
        |                 +-- sync_skipped  TERMINAL
        |                 +-- sync_complete
        |                       |
        |                       +-- [docCleanRunner]
        |                             +-- clean_failed .. TERMINAL
        |                             +-- clean_passed .. TERMINAL

Non-terminal outcomes (submitted, init_complete, sync_complete) indicate the document is still being processed. Poll the batch details endpoint to see updates as files progress through each stage.

7. Folder Cleanup Coupling

Legacy ZIP batch uploads still create folders during extraction. For batch uploads, the backend records touched folder segments in batch_folder_candidates and finalizes cleanup when the batch has terminal outcome evidence.

Accepted terminal outcomes (clean_passed, sync_skipped) keep their folder assignments. Rejected outcomes (duplicate, init_duplicate, invalid_type, failed_open, failed_read, failed_s3_upload, failed_upload, clean_failed) detach current-batch documents/uploads from auto-created candidate folders. Empty auto-created folders are then deleted only when they are not root, did not pre-exist, were not renamed, have no documents/uploads/children/chatbot scopes, and are not pending cleanup for another batch.

If outcome persistence fails before any outcome row exists, cleanup remains pending instead of marking folder_cleanup_completed_at. This keeps failed zero-outcome batches visible for retry/diagnosis and prevents cleanup from closing before rejected artifacts can be detached.