query-orchestration/scripts/manual_tests/monitoring/README.md

Document Processing Monitor

This directory contains scripts to monitor document processing status after running batch uploads.

Quick Start

1. Navigate to the manual tests directory:

   cd scripts/manual_tests
   

2. Run your batch upload:

   ./test_zip_upload2.sh
   

3. Check processing status:

   ./monitoring/check_processing.sh
   

What the Monitor Shows

The check_processing.sh script provides a post-upload analysis showing:

• Batch Status: Overall batch processing state and progress
• Document Breakdown: Document counts through each processing stage (uploaded, cleaned, extracted)
• Queue Depths: Current number of messages in each SQS queue
• Processing Analysis: Where processing stopped and why
• Next Steps: Specific solutions for common issues

Sample Output

========================================
Document Processing Status Check
========================================

[INFO] Finding most recent batch...
[SUCCESS] Found batch: 550e8400-e29b-41d4-a716-446655440000

[INFO] Batch Status:
  Status: processing
  Progress: 2/3 documents processed

[INFO] Document Breakdown:
  Total documents in DB: 3
  Uploaded to storage:   3
  Passed cleaning:       0
  Text extracted:        0

[INFO] Current Queue Depths:
  store_event:    0 messages
  document_init:  0 messages
  document_sync:  1 messages
  document_clean: 0 messages
  document_text:  0 messages
  query_sync:     0 messages
  query_runner:   0 messages

[INFO] Processing Analysis:
[WARNING] Documents blocked at sync stage: Client CanSync flag is false
  Solution: Run 'INSERT INTO clientcansync (clientid, cansync) VALUES ('[client_id]', true);'

⚠️  Processing stopped at document sync (client CanSync issue)
========================================

Common Issues and Solutions

1. Processing Stopped at Document Sync

Symptom: Documents stuck in document_sync queue
Cause: Client's CanSync flag is false or unset

About the CanSync Flag: The CanSync flag is a client-level permission control that determines whether documents for a specific client are allowed to proceed through the document synchronization stage. This is a business rule enforcement mechanism that can be used for:

• Client onboarding workflows (only allow sync after certain prerequisites)
• Compliance requirements (only sync documents for authorized clients)
• Feature gating (premium vs basic client tiers)

Default Behavior: New clients do NOT have a CanSync entry by default (NULL state), which blocks processing.

Solution:

INSERT INTO clientcansync (clientid, cansync) VALUES ('your_client_id', true);

For Demo Purposes: Always set cansync = true to allow the full document processing pipeline to run and demonstrate all stages working.

2. Processing Stopped at Document Cleaning

Symptom: Documents stuck in document_clean queue with MIME type errors Cause: S3 objects missing proper Content-Type headers Solution: Ensure MIME type detection is working in upload code

3. No Processing Activity

Symptom: Documents remain in pending status Cause: Queue runners may not be running Solution: Check docker ps and restart services if needed

Requirements

• AWS CLI (for SQS queue monitoring)
• Docker (for database queries and service log access)
• Running query-orchestration stack

Files

• check_processing.sh - Main monitoring script
• README.md - This documentation