query-orchestration/docs/ai.generated/08-operations-guide.md

# Operations Guide

## Updated Operations (September 2025)

This guide covers deployment, monitoring, and operational procedures for the enhanced platform including batch processing capabilities, comprehensive Prometheus metrics, and background task management.

## Deployment Procedures

### Local Development Deployment
```bash
# 1. Start development environment
devbox shell

# 2. Start all services with dependencies
task compose:up

# 3. Verify deployment
curl http://localhost:8080/health

# 4. View logs
docker compose -f deployments/compose.local.yaml logs -f
```

### Production Deployment

#### Prerequisites
- Container orchestration platform (Docker Swarm, Kubernetes, ECS)
- PostgreSQL 17.2+ database
- AWS account with required services
- Load balancer for API endpoints
- Monitoring infrastructure

#### Deployment Steps
```bash
# 1. Build production images
task docker:build

# 2. Push to container registry
task docker:push

# 3. Update configuration
kubectl apply -f k8s/config/

# 4. Deploy services
kubectl apply -f k8s/services/

# 5. Verify deployment
kubectl rollout status deployment/query-api
```

#### Environment-Specific Configurations

**Staging Environment**:
```yaml
# staging.env
LOG_LEVEL=DEBUG
ENABLE_OTEL=true
DB_NOSSL=false

# Staging AWS resources
AWS_REGION=us-west-2
BUCKET=doczy-staging-documents
```

**Production Environment**:
```yaml
# production.env
LOG_LEVEL=INFO
ENABLE_OTEL=true
DB_NOSSL=false

# Production AWS resources
AWS_REGION=us-east-1
BUCKET=doczy-production-documents
```

### Rolling Updates
```bash
# Update API service with zero downtime
kubectl set image deployment/query-api query-api=registry/query-api:v1.2.0

# Monitor rollout
kubectl rollout status deployment/query-api

# Rollback if needed
kubectl rollout undo deployment/query-api
```

## Monitoring and Alerting

### Enhanced Prometheus Metrics (September 2025)

The platform now provides comprehensive observability with 12 standard metric types covering all aspects of system performance and health.

### Prometheus Metrics

#### System Metrics
All services automatically expose metrics at `/metrics`:
- **HTTP requests**: Request count, duration, status codes
- **Database connections**: Pool usage, connection count, query duration
- **Queue processing**: Message count, processing time, errors
- **Business metrics**: Documents processed, queries executed, results generated
- **Batch processing**: ZIP upload metrics, batch completion rates, failed document tracking
- **Background tasks**: Task execution statistics, periodic job performance
- **Resource usage**: Memory, CPU, goroutine counts
- **Cache operations**: Hit/miss ratios, cache size, eviction rates

#### Custom Metrics Example
```go
// Counter for processed documents
var documentsProcessed = promauto.NewCounterVec(
    prometheus.CounterOpts{
        Name: "documents_processed_total",
        Help: "Total number of documents processed",
    },
    []string{"client_id", "status"},
)

// Histogram for query execution time
var queryExecutionTime = promauto.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "query_execution_duration_seconds",
        Help: "Time spent executing queries",
    },
    []string{"query_type"},
)
```

#### Key Metrics to Monitor
```promql
# API Health
up{job="query-api"}
http_requests_total{job="query-api"}
http_request_duration_seconds{job="query-api"}

# Database Health
pg_connections_active
pg_connections_max_reached
query_duration_seconds

# Queue Health
sqs_messages_visible
sqs_messages_inflight
queue_processing_duration_seconds

# Business Metrics
documents_processed_total
query_executions_total
export_operations_total
```

### Health Checks

#### Service Health Endpoints
Each service exposes a health endpoint:
```bash
# API service health
curl http://localhost:8080/health

# Individual service health
curl http://localhost:8081/health  # storeEventRunner
curl http://localhost:8082/health  # docInitRunner
# ... etc
```

#### Health Check Implementation
```go
func (h *HealthHandler) IsHealthy(ctx echo.Context) error {
    status := make(map[string]string)
    
    // Check database connectivity
    if err := h.db.Ping(ctx.Request().Context()); err != nil {
        status["database"] = "unhealthy: " + err.Error()
        return ctx.JSON(http.StatusServiceUnavailable, status)
    }
    status["database"] = "healthy"
    
    // Check queue connectivity
    if err := h.queue.Health(); err != nil {
        status["queue"] = "unhealthy: " + err.Error()
        return ctx.JSON(http.StatusServiceUnavailable, status)
    }
    status["queue"] = "healthy"
    
    return ctx.JSON(http.StatusOK, status)
}
```

#### Kubernetes Health Checks
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: query-api
spec:
  template:
    spec:
      containers:
      - name: query-api
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
```

### Alerting Rules

#### Critical Alerts
```yaml
# Service Down
- alert: ServiceDown
  expr: up == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Service {{ $labels.instance }} is down"

# High Error Rate
- alert: HighErrorRate
  expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High error rate on {{ $labels.instance }}"

# Database Connection Issues
- alert: DatabaseConnectionHigh
  expr: pg_connections_active / pg_connections_max > 0.8
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Database connection pool usage high"

# Queue Backlog
- alert: QueueBacklog
  expr: sqs_messages_visible > 1000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Queue backlog detected: {{ $value }} messages"
```

#### Business Logic Alerts
```yaml
# Document Processing Delays
- alert: DocumentProcessingDelay
  expr: increase(documents_uploaded_total[1h]) - increase(documents_processed_total[1h]) > 100
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Document processing falling behind uploads"

# Query Failures
- alert: QueryFailures
  expr: rate(query_executions_total{status="failed"}[10m]) > 0.05
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High query failure rate detected"
```

## Log Analysis and Debugging

### Structured Logging Format
```json
{
  "time": "2024-01-01T12:00:00Z",
  "level": "INFO",
  "msg": "HTTP request",
  "method": "POST",
  "uri": "/client",
  "status": 201,
  "latency": "45.123ms",
  "remote_ip": "192.168.1.100",
  "user_agent": "curl/7.68.0",
  "trace_id": "abc123def456",
  "span_id": "789ghi012jkl"
}
```

### Log Aggregation Queries

#### Common Log Searches
```bash
# Error logs across all services
level:"ERROR" | timechart span=5m count by service

# API endpoint performance
service:"queryAPI" AND uri:"/client" | stats avg(latency) by uri

# Database query performance
msg:"database query" | stats avg(duration) by query_type

# Authentication failures
msg:"authentication failed" | stats count by remote_ip

# Queue processing errors
service:"*Runner" AND level:"ERROR" | timechart span=1m count
```

#### Performance Analysis
```bash
# Slow API requests
service:"queryAPI" AND latency > 1000ms | head 100

# Database connection issues
msg:"database connection" AND level:"ERROR"

# Queue processing bottlenecks (active pipeline)
service:"doc*Runner" | stats avg(processing_time) by service
```

### Debugging Common Issues

#### Service Won't Start
```bash
# Check configuration
env | grep -E "(PGHOST|AWS_|QUEUE_)"

# Validate database connectivity
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE -c "SELECT 1"

# Test AWS credentials
aws sts get-caller-identity

# Check queue connectivity
aws sqs get-queue-attributes --queue-url $QUEUE_URL --attribute-names All
```

#### High Memory Usage
```bash
# Generate memory profile
curl http://localhost:8080/debug/pprof/heap > heap.prof
go tool pprof heap.prof

# Check for goroutine leaks
curl http://localhost:8080/debug/pprof/goroutine > goroutine.prof
go tool pprof goroutine.prof
```

#### Database Performance Issues
[link to rendered version here](https://mermaid.live/edit#pako:eNqNU9tqg0AQ_ZXBl14woX1tyEObWGpJIiSBUhBku07MEh1TXdtKyL9319RrbKkPi3Nm58yZmZ2DwWMfjTswBgOYbJHvgHEpPhB4TITqN6bUpZU1syZrhWUkL6-v4HHpzGEfeKlk0isChMzh5claWqAxhDFcnIguRi65pNhtH0mKTQ5pGH_Ce4aJwJpa27kJETLy8Au5J0WEJnAWhim41EpYZIgUm_Y4y6m1hIfXTihMrdVEuWf23F7D7U2p4lRjGPNdnftNm-h7BTrcCx_uVxWoTNMlOH0lWJY8zFIkprI1AhSUdCIEBT3kGj1n1-iv9NrZw99UVDSyqafq1l-JqiieJYm6W0d5grxabhJzTNN6HpxJFsbBcP9TYLuXLj079qJzr_1muvrBWZzXpLs2Ph9TL31DRtX2irQehD5lvscz3tLxX-3NNrby9KlvPQSXTguzcLpPMEgYSfRHhglGhEnEhK9W9GDIrRqJXlYfNywLpXE8fgM8B1iJ)

```sql
-- Check active connections
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

-- Identify slow queries
SELECT query, mean_exec_time, calls 
FROM pg_stat_statements 
ORDER BY mean_exec_time DESC 
LIMIT 10;

-- Check locks
SELECT blocked_locks.pid AS blocked_pid,
       blocked_activity.usename AS blocked_user,
       blocking_locks.pid AS blocking_pid,
       blocking_activity.usename AS blocking_user,
       blocked_activity.query AS blocked_statement,
       blocking_activity.query AS current_statement_in_blocking_process
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
```

## Backup and Recovery

### Database Backup
```bash
# Daily backup script
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="queryorchestration_backup_$DATE.sql"

pg_dump -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE \
  --no-password --compress=9 --file=$BACKUP_FILE

# Upload to S3
aws s3 cp $BACKUP_FILE s3://backups-bucket/database/

# Cleanup old local backups
find /backups -name "*.sql" -mtime +7 -delete
```

### S3 Document Backup
```bash
# Cross-region replication configuration
aws s3api put-bucket-replication \
  --bucket source-bucket \
  --replication-configuration file://replication.json

# Point-in-time recovery setup
aws s3api put-bucket-versioning \
  --bucket documents-bucket \
  --versioning-configuration Status=Enabled
```

### Disaster Recovery

#### Database Recovery
```bash
# Restore from backup
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE < backup.sql

# Point-in-time recovery (if WAL archiving enabled)
pg_basebackup -h $PGHOST -D /recovery -U postgres -v -P -W
```

#### Service Recovery
```bash
# Restart all services
kubectl rollout restart deployment

# Verify service health
kubectl get pods -l app=query-orchestration

# Check service connectivity
task health:check:all
```

## Capacity Planning

### Resource Requirements

#### Production Sizing Guidelines
| Component | CPU | Memory | Storage | Network |
|-----------|-----|--------|---------|---------|
| queryAPI | 2-4 cores | 4-8 GB | 20 GB | 1 Gbps |
| Runners (each) | 1-2 cores | 2-4 GB | 10 GB | 500 Mbps |
| Database | 4-8 cores | 16-32 GB | 1+ TB SSD | 10 Gbps |
| Total System | 16-32 cores | 64-128 GB | 2+ TB | 10+ Gbps |

#### Scaling Thresholds
- **CPU Usage**: Scale up when > 70% for 5 minutes
- **Memory Usage**: Scale up when > 80% for 5 minutes  
- **Queue Depth**: Scale runners when > 100 messages for 10 minutes
- **Database Connections**: Alert when > 80% of max connections

### Performance Baselines
```bash
# Expected performance targets
API Response Time: < 200ms (95th percentile)
Document Processing: < 5 minutes per document
Query Execution: < 30 seconds per query
Database Query Time: < 100ms (average)
Queue Processing: < 1 second per message
```

### Growth Planning
```bash
# Monthly capacity review
- Document upload volume trends
- Query execution frequency
- Storage growth rate
- API request patterns
- Resource utilization trends
```

## AWS IAM Requirements

### Required S3 Permissions

The document processing pipeline requires specific S3 permissions for file operations. Below are the minimum IAM permissions required for each service.

#### docInitRunner Service

The `docInitRunner` service requires `s3:GetObject` permission to measure file sizes during document initialization. This is used by the `HeadObject` operation to retrieve file metadata.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DocumentInitS3Permissions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:HeadObject"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}/*"
      ]
    },
    {
      "Sid": "DocumentInitBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}"
      ]
    }
  ]
}
```

**Note**: The `s3:GetObject` permission is required for the `HeadObject` API call that measures file sizes. Without this permission, document initialization will fail with an access denied error.

#### File Size Tracking (January 2026)

Documents now track file size at initialization time. The `docInitRunner` service measures file size using S3 `HeadObject` before storing documents. This requires:

- **Permission**: `s3:GetObject` on the document bucket
- **Environment**: `BUCKET` environment variable must be set
- **Error Handling**: If HeadObject fails, document creation will fail (not silently ignored)

#### Compose File Configuration

For local development with localstack, the bucket is configured via environment variables:

```yaml
# deployments/compose.local.yaml
docInitRunner:
  environment:
    BUCKET: ${BUCKET_IN}  # Maps to the "documentin" bucket
```

For production deployments, ensure the IAM role attached to the service has the required S3 permissions above.

### SQS Permissions

Queue consumer services require SQS permissions:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SQSConsumerPermissions",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:ChangeMessageVisibility"
      ],
      "Resource": [
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_init",
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_sync",
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_clean",
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_text"
      ]
    }
  ]
}
```

### Legacy Textract Permissions

The prior text extraction/query pipeline is deprecated. Keep Textract IAM permissions only if you still run legacy compatibility services in your deployment.

## Security Operations

### Security Monitoring
```bash
# Authentication failures
grep "authentication failed" /var/log/queryapi/*.log | wc -l

# Unusual API access patterns
tail -f /var/log/queryapi/access.log | grep -E "(POST|PUT|DELETE)"

# Database access monitoring
SELECT * FROM pg_stat_activity WHERE state = 'active' AND backend_type = 'client backend';
```

### Incident Response

#### Security Incident Procedure
1. **Detect**: Monitor alerts and logs for anomalies
2. **Assess**: Determine scope and impact
3. **Contain**: Isolate affected systems
4. **Investigate**: Analyze logs and system state
5. **Recover**: Restore normal operations
6. **Learn**: Document and improve procedures

#### Emergency Contacts
- **Platform Team**: platform-team@company.com
- **Security Team**: security@company.com
- **Database Team**: dba@company.com
- **AWS Support**: Enterprise support case

### Compliance and Auditing
```bash
# Audit log collection
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=query-orchestration

# Access log analysis
grep -E "(POST|PUT|DELETE)" /var/log/nginx/access.log | awk '{print $1, $7, $9}' | sort | uniq -c

# Database audit queries
SELECT * FROM pg_stat_user_tables WHERE n_tup_ins + n_tup_upd + n_tup_del > 0;
```