query-orchestration/docs/ai.generated/08-operations-guide.md

Operations Guide

Deployment Procedures

Local Development Deployment

# 1. Start development environment
devbox shell

# 2. Start all services with dependencies
task dev:up

# 3. Verify deployment
task health:check

# 4. View logs
task logs:follow

Production Deployment

Prerequisites

• Container orchestration platform (Docker Swarm, Kubernetes, ECS)
• PostgreSQL 17.2+ database
• AWS account with required services
• Load balancer for API endpoints
• Monitoring infrastructure

Deployment Steps

# 1. Build production images
task docker:build

# 2. Push to container registry
task docker:push

# 3. Update configuration
kubectl apply -f k8s/config/

# 4. Deploy services
kubectl apply -f k8s/services/

# 5. Verify deployment
kubectl rollout status deployment/query-api

Environment-Specific Configurations

Staging Environment:

# staging.env
LOG_LEVEL=DEBUG
ENABLE_OTEL=true
DB_SSL_MODE=require

# Staging AWS resources
AWS_REGION=us-west-2
S3_BUCKET=doczy-staging-documents

Production Environment:

# production.env
LOG_LEVEL=INFO
ENABLE_OTEL=true
DB_SSL_MODE=require

# Production AWS resources
AWS_REGION=us-east-1
S3_BUCKET=doczy-production-documents

Rolling Updates

# Update API service with zero downtime
kubectl set image deployment/query-api query-api=registry/query-api:v1.2.0

# Monitor rollout
kubectl rollout status deployment/query-api

# Rollback if needed
kubectl rollout undo deployment/query-api

Monitoring and Alerting

Prometheus Metrics

System Metrics

All services automatically expose metrics at /metrics:

• HTTP requests: Request count, duration, status codes
• Database connections: Pool usage, connection count, query duration
• Queue processing: Message count, processing time, errors
• Business metrics: Documents processed, queries executed, results generated

Custom Metrics Example

// Counter for processed documents
var documentsProcessed = promauto.NewCounterVec(
    prometheus.CounterOpts{
        Name: "documents_processed_total",
        Help: "Total number of documents processed",
    },
    []string{"client_id", "status"},
)

// Histogram for query execution time
var queryExecutionTime = promauto.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "query_execution_duration_seconds",
        Help: "Time spent executing queries",
    },
    []string{"query_type"},
)

Key Metrics to Monitor

# API Health
up{job="query-api"}
http_requests_total{job="query-api"}
http_request_duration_seconds{job="query-api"}

# Database Health
pg_connections_active
pg_connections_max_reached
query_duration_seconds

# Queue Health
sqs_messages_visible
sqs_messages_inflight
queue_processing_duration_seconds

# Business Metrics
documents_processed_total
query_executions_total
export_operations_total

Health Checks

Service Health Endpoints

Each service exposes a health endpoint:

# API service health
curl http://localhost:8080/health

# Individual service health
curl http://localhost:8081/health  # storeEventRunner
curl http://localhost:8082/health  # docInitRunner
# ... etc

Health Check Implementation

func (h *HealthHandler) IsHealthy(ctx echo.Context) error {
    status := make(map[string]string)
    
    // Check database connectivity
    if err := h.db.Ping(ctx.Request().Context()); err != nil {
        status["database"] = "unhealthy: " + err.Error()
        return ctx.JSON(http.StatusServiceUnavailable, status)
    }
    status["database"] = "healthy"
    
    // Check queue connectivity
    if err := h.queue.Health(); err != nil {
        status["queue"] = "unhealthy: " + err.Error()
        return ctx.JSON(http.StatusServiceUnavailable, status)
    }
    status["queue"] = "healthy"
    
    return ctx.JSON(http.StatusOK, status)
}

Kubernetes Health Checks

apiVersion: apps/v1
kind: Deployment
metadata:
  name: query-api
spec:
  template:
    spec:
      containers:
      - name: query-api
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Alerting Rules

Critical Alerts

# Service Down
- alert: ServiceDown
  expr: up == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Service {{ $labels.instance }} is down"

# High Error Rate
- alert: HighErrorRate
  expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High error rate on {{ $labels.instance }}"

# Database Connection Issues
- alert: DatabaseConnectionHigh
  expr: pg_connections_active / pg_connections_max > 0.8
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Database connection pool usage high"

# Queue Backlog
- alert: QueueBacklog
  expr: sqs_messages_visible > 1000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Queue backlog detected: {{ $value }} messages"

Business Logic Alerts

# Document Processing Delays
- alert: DocumentProcessingDelay
  expr: increase(documents_uploaded_total[1h]) - increase(documents_processed_total[1h]) > 100
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Document processing falling behind uploads"

# Query Failures
- alert: QueryFailures
  expr: rate(query_executions_total{status="failed"}[10m]) > 0.05
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High query failure rate detected"

Log Analysis and Debugging

Structured Logging Format

{
  "time": "2024-01-01T12:00:00Z",
  "level": "INFO",
  "msg": "HTTP request",
  "method": "POST",
  "uri": "/client",
  "status": 201,
  "latency": "45.123ms",
  "remote_ip": "192.168.1.100",
  "user_agent": "curl/7.68.0",
  "trace_id": "abc123def456",
  "span_id": "789ghi012jkl"
}

Log Aggregation Queries

Common Log Searches

# Error logs across all services
level:"ERROR" | timechart span=5m count by service

# API endpoint performance
service:"queryAPI" AND uri:"/client" | stats avg(latency) by uri

# Database query performance
msg:"database query" | stats avg(duration) by query_type

# Authentication failures
msg:"authentication failed" | stats count by remote_ip

# Queue processing errors
service:"*Runner" AND level:"ERROR" | timechart span=1m count

Performance Analysis

# Slow API requests
service:"queryAPI" AND latency > 1000ms | head 100

# Database connection issues
msg:"database connection" AND level:"ERROR"

# Queue processing bottlenecks
service:"queryRunner" | stats avg(processing_time) by query_type

Debugging Common Issues

Service Won't Start

# Check configuration
env | grep -E "(PGHOST|AWS_|QUEUE_)"

# Validate database connectivity
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE -c "SELECT 1"

# Test AWS credentials
aws sts get-caller-identity

# Check queue connectivity
aws sqs get-queue-attributes --queue-url $QUEUE_URL --attribute-names All

High Memory Usage

# Generate memory profile
curl http://localhost:8080/debug/pprof/heap > heap.prof
go tool pprof heap.prof

# Check for goroutine leaks
curl http://localhost:8080/debug/pprof/goroutine > goroutine.prof
go tool pprof goroutine.prof

Database Performance Issues

link to rendered version here

-- Check active connections
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

-- Identify slow queries
SELECT query, mean_exec_time, calls 
FROM pg_stat_statements 
ORDER BY mean_exec_time DESC 
LIMIT 10;

-- Check locks
SELECT blocked_locks.pid AS blocked_pid,
       blocked_activity.usename AS blocked_user,
       blocking_locks.pid AS blocking_pid,
       blocking_activity.usename AS blocking_user,
       blocked_activity.query AS blocked_statement,
       blocking_activity.query AS current_statement_in_blocking_process
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;

Backup and Recovery

Database Backup

# Daily backup script
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="queryorchestration_backup_$DATE.sql"

pg_dump -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE \
  --no-password --compress=9 --file=$BACKUP_FILE

# Upload to S3
aws s3 cp $BACKUP_FILE s3://backups-bucket/database/

# Cleanup old local backups
find /backups -name "*.sql" -mtime +7 -delete

S3 Document Backup

# Cross-region replication configuration
aws s3api put-bucket-replication \
  --bucket source-bucket \
  --replication-configuration file://replication.json

# Point-in-time recovery setup
aws s3api put-bucket-versioning \
  --bucket documents-bucket \
  --versioning-configuration Status=Enabled

Disaster Recovery

Database Recovery

# Restore from backup
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE < backup.sql

# Point-in-time recovery (if WAL archiving enabled)
pg_basebackup -h $PGHOST -D /recovery -U postgres -v -P -W

Service Recovery

# Restart all services
kubectl rollout restart deployment

# Verify service health
kubectl get pods -l app=query-orchestration

# Check service connectivity
task health:check:all

Capacity Planning

Resource Requirements

Production Sizing Guidelines

Component	CPU	Memory	Storage	Network
queryAPI	2-4 cores	4-8 GB	20 GB	1 Gbps
Runners (each)	1-2 cores	2-4 GB	10 GB	500 Mbps
Database	4-8 cores	16-32 GB	1+ TB SSD	10 Gbps
Total System	16-32 cores	64-128 GB	2+ TB	10+ Gbps

Scaling Thresholds

• CPU Usage: Scale up when > 70% for 5 minutes
• Memory Usage: Scale up when > 80% for 5 minutes
• Queue Depth: Scale runners when > 100 messages for 10 minutes
• Database Connections: Alert when > 80% of max connections

Performance Baselines

# Expected performance targets
API Response Time: < 200ms (95th percentile)
Document Processing: < 5 minutes per document
Query Execution: < 30 seconds per query
Database Query Time: < 100ms (average)
Queue Processing: < 1 second per message

Growth Planning

# Monthly capacity review
- Document upload volume trends
- Query execution frequency
- Storage growth rate
- API request patterns
- Resource utilization trends

Security Operations

Security Monitoring

# Authentication failures
grep "authentication failed" /var/log/queryapi/*.log | wc -l

# Unusual API access patterns
tail -f /var/log/queryapi/access.log | grep -E "(POST|PUT|DELETE)"

# Database access monitoring
SELECT * FROM pg_stat_activity WHERE state = 'active' AND backend_type = 'client backend';

Incident Response

Security Incident Procedure

1. Detect: Monitor alerts and logs for anomalies
2. Assess: Determine scope and impact
3. Contain: Isolate affected systems
4. Investigate: Analyze logs and system state
5. Recover: Restore normal operations
6. Learn: Document and improve procedures

Emergency Contacts

• Platform Team: platform-team@company.com
• Security Team: security@company.com
• Database Team: dba@company.com
• AWS Support: Enterprise support case

Compliance and Auditing

# Audit log collection
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=query-orchestration

# Access log analysis
grep -E "(POST|PUT|DELETE)" /var/log/nginx/access.log | awk '{print $1, $7, $9}' | sort | uniq -c

# Database audit queries
SELECT * FROM pg_stat_user_tables WHERE n_tup_ins + n_tup_upd + n_tup_del > 0;