query-orchestration/docs/ai.generated/01-system-overview.md

# System Overview

## DoczyAI Query Orchestration Platform

### Purpose and Domain
DoczyAI is a cloud-native document processing and query orchestration platform designed to extract structured information from documents (primarily PDFs) using configurable, hierarchical queries. The system serves as a multi-tenant platform where organizations can upload documents, define custom data extraction rules, and receive structured results for business intelligence and analysis.

### Core Business Capabilities
- **Multi-tenant Document Processing**: Isolated processing environments per client
- **Configurable Query System**: Custom data extraction rules with dependency management
- **Automated Processing Pipeline**: Seamless flow from document upload to structured output
- **Version Management**: Complete versioning for queries and configurations
- **Export and Integration**: Structured data export capabilities for downstream systems

### Technology Stack
- **Language**: Go 1.24.0
- **Web Framework**: Echo v4 with OpenAPI code generation
- **Database**: PostgreSQL 17.2 with SQLC for type-safe queries
- **Messaging**: AWS SQS for event-driven microservices
- **Storage**: AWS S3 for document storage
- **Text Processing**: AWS Textract for OCR and text extraction
- **Authentication**: AWS Cognito with JWT-based authorization
- **Monitoring**: Prometheus metrics and OpenTelemetry tracing
- **Container Platform**: Docker with multi-stage builds

## Architecture Overview

### System Architecture Pattern
The system follows an **event-driven microservices architecture** with the following characteristics:
- **12 distinct services**: 1 HTTP API + 8 queue-based runners + 3 utility services
- **Asynchronous communication**: SQS queues between services
- **State management**: PostgreSQL as central data store
- **External service integration**: AWS managed services for cloud capabilities

### High-Level Component Diagram

[link to rendered version here](https://mermaid.live/edit#pako:eNqFlGtr2zAUhv-K8KcOVtYtG0vHKCSyKYa2sSM3HTilKM5JYnCkTpe2Wel_ny4OlZ1C_MGWjp5z0asjv0YVX0L0C0VrQR83qBjPGTKP1AtvmEfJiwLBaINwUwNTch55xD63aXkHC5Qyg6xoBffvS6MsfcBXaXJTlGa4d24BYEs_OMhm2Uuq4JnuwkS5sZd_NYidGfxeiC8XGRcKDc-GZ0dDxrzSW5McZYJXIGXN1iirH6GpGYQ5SFJKxQUkTwaeasZAuEwmydd7dHp6geK0XPIqZXVv-Vuw7zj1KLEo2bGqiw5ClHgUWxQ3QFmX_R6y2LOFZQt46VXwI0QLh-bEC3ZYw88Azn0N-dTDXXB4VFsbfCM4q_9RVXOGCIin2ogc6opJWbnTP6zkPKxk1hY8AyFNrAP6_PhRp2wlqFRCV0oL-LCaeFyeZFyqtQCSX7nQMVV0QSV8Cqohg_JkdEfM1yHENAZddwkjsCNy4pBcgwYZAEXyp5iOcOEoe2SCVsqhdoISbzA7DXzw5PImLSbOBfO16TTuPEZabYyCdUV7Dtl0cl2axt6CAbR08DW3fsL0-ceC3foWtXfKG97vam_Bv-3cd9-4ZyCDviEnPUu7ozAeSbps3I2-vxb7Ke6kart7L26breuSTztT_8akV-GsZ2i1aJqHfd-4datw9BlFWxBbWi_Nv_I1Mlpv3V9zCSuqGxW9vf0HHgWInQ==)

```mermaid
graph TB
    subgraph "External Clients"
        UI[Web Interface]
        API_CLIENT[API Clients]
    end
    
    subgraph "API Gateway"
        QAPI[queryAPI<br/>Port 8080]
    end
    
    subgraph "Document Processing Pipeline"
        SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
        DI --> DS[docSyncRunner<br/>8083]
        DS --> DC[docCleanRunner<br/>8084]
        DC --> DT[docTextRunner<br/>8085]
        DT --> QS[querySyncRunner<br/>8087]
        QS --> QR[queryRunner<br/>8088]
    end
    
    subgraph "Synchronization Services"
        CS[clientSyncRunner<br/>8089]
        QVS[queryVersionSyncRunner<br/>8090]
    end
    
    subgraph "Infrastructure Services"
        DB[(PostgreSQL<br/>Database)]
        S3[(AWS S3<br/>Storage)]
        SQS[AWS SQS<br/>Queues]
        TEXTRACT[AWS Textract<br/>Text Extraction]
        COGNITO[AWS Cognito<br/>Authentication]
        PROM[Prometheus<br/>Monitoring]
    end
    
    UI --> QAPI
    API_CLIENT --> QAPI
    
    QAPI --> DB
    QAPI --> S3
    QAPI --> SQS
    QAPI --> COGNITO
    
    SE --> SQS
    DI --> DB
    DS --> DB
    DC --> S3
    DT --> TEXTRACT
    QS --> DB
    QR --> DB
    
    CS --> SQS
    QVS --> SQS
    
    All_Services --> PROM
```

## Deployment Architecture

### Infrastructure Requirements
- **Compute**: Container orchestration platform (Docker/Kubernetes)
- **Database**: PostgreSQL 17.2+ with connection pooling
- **AWS Services**: S3, SQS, Textract, Cognito services
- **Monitoring**: Prometheus-compatible metrics collection
- **Networking**: Load balancer for API endpoints

### Scalability Considerations
- **Horizontal Scaling**: All runners can be scaled independently based on queue depth
- **Database Scaling**: Connection pooling with configurable pool sizes
- **Storage Scaling**: S3 provides virtually unlimited document storage
- **Processing Scaling**: Auto-scaling based on SQS queue metrics

### Security Architecture
- **Authentication**: OAuth2 with AWS Cognito integration
- **Authorization**: Role-based access control with group-based permissions
- **Data Protection**: Encrypted storage (S3 server-side encryption)
- **Network Security**: VPC isolation and security groups
- **API Security**: JWT validation and OpenAPI request validation

## Integration Points

### External System Dependencies
1. **AWS S3**: Document storage and retrieval
2. **AWS SQS**: Inter-service messaging and event processing
3. **AWS Textract**: OCR and text extraction from documents
4. **AWS Cognito**: User authentication and session management
5. **PostgreSQL**: Central data persistence and query processing

### API Integration
- **REST API**: OpenAPI 3.0.3 specification with Swagger UI
- **Authentication**: Bearer token or OAuth2 flows
- **Rate Limiting**: 100 requests per 60-second window
- **Content Types**: JSON for API, multipart/form-data for uploads

### Performance Characteristics
- **Throughput**: Designed for high-volume document processing
- **Latency**: Sub-second API response times for most operations
- **Availability**: Distributed architecture supports high availability
- **Durability**: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively