query-orchestration/docs/ai.generated/08-operations-guide.md

Operations Guide

Updated Operations (September 2025)

This guide covers deployment, monitoring, and operational procedures for the enhanced platform including batch processing capabilities, comprehensive Prometheus metrics, and background task management.

Deployment Procedures

Local Development Deployment

# 1. Start development environment
devbox shell

# 2. Start all services with dependencies
task compose:up

# 3. Verify deployment
curl http://localhost:8080/health

# 4. View logs
docker compose -f deployments/compose.local.yaml logs -f

Production Deployment

Prerequisites

• Container orchestration platform (Docker Swarm, Kubernetes, ECS)
• PostgreSQL 17.2+ database
• AWS account with required services
• Load balancer for API endpoints
• Monitoring infrastructure

Deployment Steps

# 1. Build production images
task docker:build

# 2. Push to container registry
task docker:push

# 3. Update configuration
kubectl apply -f k8s/config/

# 4. Deploy services
kubectl apply -f k8s/services/

# 5. Verify deployment
kubectl rollout status deployment/query-api

Environment-Specific Configurations

Staging Environment:

# staging.env
LOG_LEVEL=DEBUG
ENABLE_OTEL=true
DB_NOSSL=false

# Staging AWS resources
AWS_REGION=us-west-2
BUCKET=doczy-staging-documents

Production Environment:

# production.env
LOG_LEVEL=INFO
ENABLE_OTEL=true
DB_NOSSL=false

# Production AWS resources
AWS_REGION=us-east-1
BUCKET=doczy-production-documents

Rolling Updates

# Update API service with zero downtime
kubectl set image deployment/query-api query-api=registry/query-api:v1.2.0

# Monitor rollout
kubectl rollout status deployment/query-api

# Rollback if needed
kubectl rollout undo deployment/query-api

Monitoring and Alerting

Enhanced Prometheus Metrics (September 2025)

The platform now provides comprehensive observability with 12 standard metric types covering all aspects of system performance and health.

Prometheus Metrics

System Metrics

All services automatically expose metrics at /metrics:

• HTTP requests: Request count, duration, status codes
• Database connections: Pool usage, connection count, query duration
• Queue processing: Message count, processing time, errors
• Business metrics: Documents processed, queries executed, results generated
• Batch processing: ZIP upload metrics, batch completion rates, failed document tracking
• Background tasks: Task execution statistics, periodic job performance
• Resource usage: Memory, CPU, goroutine counts
• Cache operations: Hit/miss ratios, cache size, eviction rates

Custom Metrics Example

// Counter for processed documents
var documentsProcessed = promauto.NewCounterVec(
    prometheus.CounterOpts{
        Name: "documents_processed_total",
        Help: "Total number of documents processed",
    },
    []string{"client_id", "status"},
)

// Histogram for query execution time
var queryExecutionTime = promauto.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "query_execution_duration_seconds",
        Help: "Time spent executing queries",
    },
    []string{"query_type"},
)

Key Metrics to Monitor

# API Health
up{job="query-api"}
http_requests_total{job="query-api"}
http_request_duration_seconds{job="query-api"}

# Database Health
pg_connections_active
pg_connections_max_reached
query_duration_seconds

# Queue Health
sqs_messages_visible
sqs_messages_inflight
queue_processing_duration_seconds

# Business Metrics
documents_processed_total
query_executions_total
export_operations_total

Health Checks

Service Health Endpoints

Each service exposes a health endpoint:

# API service health
curl http://localhost:8080/health

# Individual service health
curl http://localhost:8081/health  # storeEventRunner
curl http://localhost:8082/health  # docInitRunner
# ... etc

The queryAPI health response includes version, buildTime, and commit fields. When COGNITO_ENVIRONMENT_ID is configured, an additional cognitoEnvironmentId field is included in the response, allowing operators to verify which environment the server instance is scoped to:

{
  "status": "healthy",
  "version": "1.0.0",
  "buildTime": "2025-01-15T10:00:00Z",
  "commit": "abc1234",
  "cognitoEnvironmentId": "acmehealth"
}

Health Check Implementation

func (h *HealthHandler) IsHealthy(ctx echo.Context) error {
    status := make(map[string]string)
    
    // Check database connectivity
    if err := h.db.Ping(ctx.Request().Context()); err != nil {
        status["database"] = "unhealthy: " + err.Error()
        return ctx.JSON(http.StatusServiceUnavailable, status)
    }
    status["database"] = "healthy"
    
    // Check queue connectivity
    if err := h.queue.Health(); err != nil {
        status["queue"] = "unhealthy: " + err.Error()
        return ctx.JSON(http.StatusServiceUnavailable, status)
    }
    status["queue"] = "healthy"
    
    return ctx.JSON(http.StatusOK, status)
}

Kubernetes Health Checks

apiVersion: apps/v1
kind: Deployment
metadata:
  name: query-api
spec:
  template:
    spec:
      containers:
      - name: query-api
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Alerting Rules

Critical Alerts

# Service Down
- alert: ServiceDown
  expr: up == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Service {{ $labels.instance }} is down"

# High Error Rate
- alert: HighErrorRate
  expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High error rate on {{ $labels.instance }}"

# Database Connection Issues
- alert: DatabaseConnectionHigh
  expr: pg_connections_active / pg_connections_max > 0.8
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Database connection pool usage high"

# Queue Backlog
- alert: QueueBacklog
  expr: sqs_messages_visible > 1000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Queue backlog detected: {{ $value }} messages"

Business Logic Alerts

# Document Processing Delays
- alert: DocumentProcessingDelay
  expr: increase(documents_uploaded_total[1h]) - increase(documents_processed_total[1h]) > 100
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Document processing falling behind uploads"

# Query Failures
- alert: QueryFailures
  expr: rate(query_executions_total{status="failed"}[10m]) > 0.05
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High query failure rate detected"

Log Analysis and Debugging

Structured Logging Format

{
  "time": "2024-01-01T12:00:00Z",
  "level": "INFO",
  "msg": "HTTP request",
  "method": "POST",
  "uri": "/client",
  "status": 201,
  "latency": "45.123ms",
  "remote_ip": "192.168.1.100",
  "user_agent": "curl/7.68.0",
  "trace_id": "abc123def456",
  "span_id": "789ghi012jkl"
}

Log Aggregation Queries

Common Log Searches

# Error logs across all services
level:"ERROR" | timechart span=5m count by service

# API endpoint performance
service:"queryAPI" AND uri:"/client" | stats avg(latency) by uri

# Database query performance
msg:"database query" | stats avg(duration) by query_type

# Authentication failures
msg:"authentication failed" | stats count by remote_ip

# Queue processing errors
service:"*Runner" AND level:"ERROR" | timechart span=1m count

Performance Analysis

# Slow API requests
service:"queryAPI" AND latency > 1000ms | head 100

# Database connection issues
msg:"database connection" AND level:"ERROR"

# Queue processing bottlenecks (active pipeline)
service:"doc*Runner" | stats avg(processing_time) by service

Debugging Common Issues

Service Won't Start

# Check configuration
env | grep -E "(PGHOST|AWS_|QUEUE_)"

# Validate database connectivity
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE -c "SELECT 1"

# Test AWS credentials
aws sts get-caller-identity

# Check queue connectivity
aws sqs get-queue-attributes --queue-url $QUEUE_URL --attribute-names All

High Memory Usage

# Generate memory profile
curl http://localhost:8080/debug/pprof/heap > heap.prof
go tool pprof heap.prof

# Check for goroutine leaks
curl http://localhost:8080/debug/pprof/goroutine > goroutine.prof
go tool pprof goroutine.prof

Database Performance Issues

link to rendered version here

-- Check active connections
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

-- Identify slow queries
SELECT query, mean_exec_time, calls 
FROM pg_stat_statements 
ORDER BY mean_exec_time DESC 
LIMIT 10;

-- Check locks
SELECT blocked_locks.pid AS blocked_pid,
       blocked_activity.usename AS blocked_user,
       blocking_locks.pid AS blocking_pid,
       blocking_activity.usename AS blocking_user,
       blocked_activity.query AS blocked_statement,
       blocking_activity.query AS current_statement_in_blocking_process
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;

Backup and Recovery

Database Backup

# Daily backup script
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="queryorchestration_backup_$DATE.sql"

pg_dump -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE \
  --no-password --compress=9 --file=$BACKUP_FILE

# Upload to S3
aws s3 cp $BACKUP_FILE s3://backups-bucket/database/

# Cleanup old local backups
find /backups -name "*.sql" -mtime +7 -delete

S3 Document Backup

# Cross-region replication configuration
aws s3api put-bucket-replication \
  --bucket source-bucket \
  --replication-configuration file://replication.json

# Point-in-time recovery setup
aws s3api put-bucket-versioning \
  --bucket documents-bucket \
  --versioning-configuration Status=Enabled

Disaster Recovery

Database Recovery

# Restore from backup
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE < backup.sql

# Point-in-time recovery (if WAL archiving enabled)
pg_basebackup -h $PGHOST -D /recovery -U postgres -v -P -W

Service Recovery

# Restart all services
kubectl rollout restart deployment

# Verify service health
kubectl get pods -l app=query-orchestration

# Check service connectivity
task health:check:all

Capacity Planning

Resource Requirements

Production Sizing Guidelines

Component	CPU	Memory	Storage	Network
queryAPI	2-4 cores	4-8 GB	20 GB	1 Gbps
Runners (each)	1-2 cores	2-4 GB	10 GB	500 Mbps
Database	4-8 cores	16-32 GB	1+ TB SSD	10 Gbps
Total System	16-32 cores	64-128 GB	2+ TB	10+ Gbps

Scaling Thresholds

• CPU Usage: Scale up when > 70% for 5 minutes
• Memory Usage: Scale up when > 80% for 5 minutes
• Queue Depth: Scale runners when > 100 messages for 10 minutes
• Database Connections: Alert when > 80% of max connections

Performance Baselines

# Expected performance targets
API Response Time: < 200ms (95th percentile)
Document Processing: < 5 minutes per document
Query Execution: < 30 seconds per query
Database Query Time: < 100ms (average)
Queue Processing: < 1 second per message

Growth Planning

# Monthly capacity review
- Document upload volume trends
- Query execution frequency
- Storage growth rate
- API request patterns
- Resource utilization trends

AWS IAM Requirements

Required S3 Permissions

The document processing pipeline requires specific S3 permissions for file operations. Below are the minimum IAM permissions required for each service.

docInitRunner Service

The docInitRunner service requires s3:GetObject permission to measure file sizes during document initialization. This is used by the HeadObject operation to retrieve file metadata.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DocumentInitS3Permissions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:HeadObject"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}/*"
      ]
    },
    {
      "Sid": "DocumentInitBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}"
      ]
    }
  ]
}

Note: The s3:GetObject permission is required for the HeadObject API call that measures file sizes. Without this permission, document initialization will fail with an access denied error.

File Size Tracking (January 2026)

Documents now track file size at initialization time. The docInitRunner service measures file size using S3 HeadObject before storing documents. This requires:

• Permission: s3:GetObject on the document bucket
• Environment: BUCKET environment variable must be set
• Error Handling: If HeadObject fails, document creation will fail (not silently ignored)

Compose File Configuration

For local development with localstack, the bucket is configured via environment variables:

# deployments/compose.local.yaml
docInitRunner:
  environment:
    BUCKET: ${BUCKET_IN}  # Maps to the "documentin" bucket

For production deployments, ensure the IAM role attached to the service has the required S3 permissions above.

SQS Permissions

Queue consumer services require SQS permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SQSConsumerPermissions",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:ChangeMessageVisibility"
      ],
      "Resource": [
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_init",
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_sync",
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_clean",
        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_text"
      ]
    }
  ]
}

Cognito Permissions (Environment Isolation)

When COGNITO_ENVIRONMENT_ID is configured, the queryAPI service automatically registers the custom:environment_id attribute on the Cognito user pool at startup. This requires additional IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CognitoEnvironmentAttributeRegistration",
      "Effect": "Allow",
      "Action": [
        "cognito-idp:DescribeUserPool",
        "cognito-idp:AddCustomAttributes"
      ],
      "Resource": [
        "arn:aws:cognito-idp:${REGION}:${ACCOUNT_ID}:userpool/${USER_POOL_ID}"
      ]
    }
  ]
}

The DescribeUserPool permission is used to check if the attribute already exists. The AddCustomAttributes permission is only used when the attribute needs to be registered for the first time. Once registered, the attribute persists permanently on the pool (Cognito custom attributes cannot be removed).

Legacy Textract Permissions

The prior text extraction/query pipeline is deprecated. Keep Textract IAM permissions only if you still run legacy compatibility services in your deployment.

Security Operations

Security Monitoring

# Authentication failures
grep "authentication failed" /var/log/queryapi/*.log | wc -l

# Unusual API access patterns
tail -f /var/log/queryapi/access.log | grep -E "(POST|PUT|DELETE)"

# Database access monitoring
SELECT * FROM pg_stat_activity WHERE state = 'active' AND backend_type = 'client backend';

Incident Response

Security Incident Procedure

1. Detect: Monitor alerts and logs for anomalies
2. Assess: Determine scope and impact
3. Contain: Isolate affected systems
4. Investigate: Analyze logs and system state
5. Recover: Restore normal operations
6. Learn: Document and improve procedures

Emergency Contacts

• Platform Team: platform-team@company.com
• Security Team: security@company.com
• Database Team: dba@company.com
• AWS Support: Enterprise support case

Compliance and Auditing

# Audit log collection
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=query-orchestration

# Access log analysis
grep -E "(POST|PUT|DELETE)" /var/log/nginx/access.log | awk '{print $1, $7, $9}' | sort | uniq -c

# Database audit queries
SELECT * FROM pg_stat_user_tables WHERE n_tup_ins + n_tup_upd + n_tup_del > 0;