# Document Processing Monitor

This directory contains scripts to monitor document processing status after running batch uploads.

## Quick Start

1. Navigate to the manual tests directory:
   ```bash
   cd scripts/manual_tests
   ```

2. Run your batch upload:
   ```bash
   ./test_zip_upload2.sh
   ```

3. Check processing status:
   ```bash
   ./monitoring/check_processing.sh
   ```

## What the Monitor Shows

The `check_processing.sh` script provides a post-upload analysis showing:

- **Batch Status**: Overall batch processing state and progress
- **Document Breakdown**: Document counts through each processing stage (uploaded, cleaned, extracted)
- **Queue Depths**: Current number of messages in each SQS queue
- **Processing Analysis**: Where processing stopped and why
- **Next Steps**: Specific solutions for common issues

## Sample Output

```
========================================
Document Processing Status Check
========================================

[INFO] Finding most recent batch...
[SUCCESS] Found batch: 550e8400-e29b-41d4-a716-446655440000

[INFO] Batch Status:
  Status: processing
  Progress: 2/3 documents processed

[INFO] Document Breakdown:
  Total documents in DB: 3
  Uploaded to storage:   3
  Passed cleaning:       0
  Text extracted:        0

[INFO] Current Queue Depths:
  store_event:    0 messages
  document_init:  0 messages
  document_sync:  1 messages
  document_clean: 0 messages
  document_text:  0 messages
  query_sync:     0 messages
  query_runner:   0 messages

[INFO] Processing Analysis:
[WARNING] Documents blocked at sync stage: Client CanSync flag is false
  Solution: Run 'INSERT INTO clientcansync (clientid, cansync) VALUES ('[client_id]', true);'

⚠️  Processing stopped at document sync (client CanSync issue)
========================================
```

## Common Issues and Solutions

### 1. Processing Stopped at Document Sync
**Symptom**: Documents stuck in `document_sync` queue  
**Cause**: Client's CanSync flag is false or unset

**About the CanSync Flag**:
The CanSync flag is a client-level permission control that determines whether documents for a specific client are allowed to proceed through the document synchronization stage. This is a business rule enforcement mechanism that can be used for:
- Client onboarding workflows (only allow sync after certain prerequisites)
- Compliance requirements (only sync documents for authorized clients) 
- Feature gating (premium vs basic client tiers)

**Default Behavior**: New clients do NOT have a CanSync entry by default (NULL state), which blocks processing.

**Solution**: 
```sql
INSERT INTO clientcansync (clientid, cansync) VALUES ('your_client_id', true);
```

**For Demo Purposes**: Always set `cansync = true` to allow the full document processing pipeline to run and demonstrate all stages working.

### 2. Processing Stopped at Document Cleaning
**Symptom**: Documents stuck in `document_clean` queue with MIME type errors
**Cause**: S3 objects missing proper Content-Type headers
**Solution**: Ensure MIME type detection is working in upload code

### 3. No Processing Activity
**Symptom**: Documents remain in `pending` status
**Cause**: Queue runners may not be running
**Solution**: Check `docker ps` and restart services if needed

## Requirements

- AWS CLI (for SQS queue monitoring)
- Docker (for database queries and service log access)
- Running query-orchestration stack

## Files

- `check_processing.sh` - Main monitoring script
- `README.md` - This documentation