109 lines
3.4 KiB
Markdown
109 lines
3.4 KiB
Markdown
|
# Document Processing Monitor
|
||
|
|
|
||
|
|
This directory contains scripts to monitor document processing status after running batch uploads.
|
||
|
|
|
||
|
|
## Quick Start
|
||
|
|
|
||
|
|
1. Navigate to the manual tests directory:
|
||
|
|
```bash
|
||
|
|
cd scripts/manual_tests
|
||
|
|
```
|
||
|
|
|
||
|
|
2. Run your batch upload:
|
||
|
|
```bash
|
||
|
|
./test_zip_upload2.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
3. Check processing status:
|
||
|
|
```bash
|
||
|
|
./monitoring/check_processing.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
## What the Monitor Shows
|
||
|
|
|
||
|
|
The `check_processing.sh` script provides a post-upload analysis showing:
|
||
|
|
|
||
|
|
- **Batch Status**: Overall batch processing state and progress
|
||
|
|
- **Document Breakdown**: Document counts through each processing stage (uploaded, cleaned, extracted)
|
||
|
|
- **Queue Depths**: Current number of messages in each SQS queue
|
||
|
|
- **Processing Analysis**: Where processing stopped and why
|
||
|
|
- **Next Steps**: Specific solutions for common issues
|
||
|
|
|
||
|
|
## Sample Output
|
||
|
|
|
||
|
|
```
|
||
|
|
========================================
|
||
|
|
Document Processing Status Check
|
||
|
|
========================================
|
||
|
|
|
||
|
|
[INFO] Finding most recent batch...
|
||
|
|
[SUCCESS] Found batch: 550e8400-e29b-41d4-a716-446655440000
|
||
|
|
|
||
|
|
[INFO] Batch Status:
|
||
|
|
Status: processing
|
||
|
|
Progress: 2/3 documents processed
|
||
|
|
|
||
|
|
[INFO] Document Breakdown:
|
||
|
|
Total documents in DB: 3
|
||
|
|
Uploaded to storage: 3
|
||
|
|
Passed cleaning: 0
|
||
|
|
Text extracted: 0
|
||
|
|
|
||
|
|
[INFO] Current Queue Depths:
|
||
|
|
store_event: 0 messages
|
||
|
|
document_init: 0 messages
|
||
|
|
document_sync: 1 messages
|
||
|
|
document_clean: 0 messages
|
||
|
|
document_text: 0 messages
|
||
|
|
query_sync: 0 messages
|
||
|
|
query_runner: 0 messages
|
||
|
|
|
||
|
|
[INFO] Processing Analysis:
|
||
|
|
[WARNING] Documents blocked at sync stage: Client CanSync flag is false
|
||
|
|
Solution: Run 'INSERT INTO clientcansync (clientid, cansync) VALUES ('[client_id]', true);'
|
||
|
|
|
||
|
|
⚠️ Processing stopped at document sync (client CanSync issue)
|
||
|
|
========================================
|
||
|
|
```
|
||
|
|
|
||
|
|
## Common Issues and Solutions
|
||
|
|
|
||
|
|
### 1. Processing Stopped at Document Sync
|
||
|
|
**Symptom**: Documents stuck in `document_sync` queue
|
||
|
|
**Cause**: Client's CanSync flag is false or unset
|
||
|
|
|
||
|
|
**About the CanSync Flag**:
|
||
|
|
The CanSync flag is a client-level permission control that determines whether documents for a specific client are allowed to proceed through the document synchronization stage. This is a business rule enforcement mechanism that can be used for:
|
||
|
|
- Client onboarding workflows (only allow sync after certain prerequisites)
|
||
|
|
- Compliance requirements (only sync documents for authorized clients)
|
||
|
|
- Feature gating (premium vs basic client tiers)
|
||
|
|
|
||
|
|
**Default Behavior**: New clients do NOT have a CanSync entry by default (NULL state), which blocks processing.
|
||
|
|
|
||
|
|
**Solution**:
|
||
|
|
```sql
|
||
|
|
INSERT INTO clientcansync (clientid, cansync) VALUES ('your_client_id', true);
|
||
|
|
```
|
||
|
|
|
||
|
|
**For Demo Purposes**: Always set `cansync = true` to allow the full document processing pipeline to run and demonstrate all stages working.
|
||
|
|
|
||
|
|
### 2. Processing Stopped at Document Cleaning
|
||
|
|
**Symptom**: Documents stuck in `document_clean` queue with MIME type errors
|
||
|
|
**Cause**: S3 objects missing proper Content-Type headers
|
||
|
|
**Solution**: Ensure MIME type detection is working in upload code
|
||
|
|
|
||
|
|
### 3. No Processing Activity
|
||
|
|
**Symptom**: Documents remain in `pending` status
|
||
|
|
**Cause**: Queue runners may not be running
|
||
|
|
**Solution**: Check `docker ps` and restart services if needed
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
|
||
|
|
- AWS CLI (for SQS queue monitoring)
|
||
|
|
- Docker (for database queries and service log access)
|
||
|
|
- Running query-orchestration stack
|
||
|
|
|
||
|
|
## Files
|
||
|
|
|
||
|
|
- `check_processing.sh` - Main monitoring script
|
||
|
|
- `README.md` - This documentation
|