query-orchestration/docs/ai.generated/01-system-overview.md

# System Overview

## DoczyAI Document Processing Platform

### Purpose and Domain
DoczyAI is a cloud-native document processing platform for multi-tenant PDF ingestion and processing. Organizations can upload and organize documents with folders and labels, and store structured field extraction results for downstream analysis.

### Core Business Capabilities
- **Multi-tenant Document Processing**: Isolated processing environments per client
- **Batch Document Processing**: ZIP archive uploads for processing multiple documents simultaneously
- **Automated Processing Pipeline**: Seamless flow from document upload to structured output
- **Version Management**: Complete versioning for configurations
- **Export and Integration**: Structured data export capabilities for downstream systems
- **Enhanced Observability**: Comprehensive metrics and monitoring with Prometheus integration

### Technology Stack
- **Language**: Go 1.24.3
- **Web Framework**: Echo v4 with OpenAPI code generation
- **Database**: PostgreSQL 17.2 with SQLC for type-safe queries
- **Messaging**: AWS SQS for event-driven microservices
- **Storage**: AWS S3 for document storage
- **Authentication**: AWS Cognito with JWT-based authorization
- **Monitoring**: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
- **Authorization**: Permit.io RBAC integration for fine-grained permissions
- **Container Platform**: Docker with multi-stage builds

## Architecture Overview

### System Architecture Pattern
The system follows an **event-driven microservices architecture** with the following characteristics:
- **Active services**: 1 HTTP API + 5 active queue runners
- **Compatibility services**: Additional deprecated runners may remain in some deployments
- **Asynchronous communication**: SQS queues between services with batch processing capabilities
- **Background Task Framework**: Comprehensive background task execution with periodic scheduling
- **State management**: PostgreSQL as central data store with enhanced schema for batch operations
- **External service integration**: AWS managed services for cloud capabilities

### High-Level Component Diagram

[link to rendered version here](https://mermaid.live/edit#pako:eNqFlGtr2zAUhv-K8KcOVtYtG0vHKCSyKYa2sSM3HTilKM5JYnCkTpe2Wel_ny4OlZ1C_MGWjp5z0asjv0YVX0L0C0VrQR83qBjPGTKP1AtvmEfJiwLBaINwUwNTch55xD63aXkHC5Qyg6xoBffvS6MsfcBXaXJTlGa4d24BYEs_OMhm2Uuq4JnuwkS5sZd_NYidGfxeiC8XGRcKDc-GZ0dDxrzSW5McZYJXIGXN1iirH6GpGYQ5SFJKxQUkTwaeasZAuEwmydd7dHp6geK0XPIqZXVv-Vuw7zj1KLEo2bGqiw5ClHgUWxQ3QFmX_R6y2LOFZQt46VXwI0QLh-bEC3ZYw88Azn0N-dTDXXB4VFsbfCM4q_9RVXOGCIin2ogc6opJWbnTP6zkPKxk1hY8AyFNrAP6_PhRp2wlqFRCV0oL-LCaeFyeZFyqtQCSX7nQMVV0QSV8Cqohg_JkdEfM1yHENAZddwkjsCNy4pBcgwYZAEXyp5iOcOEoe2SCVsqhdoISbzA7DXzw5PImLSbOBfO16TTuPEZabYyCdUV7Dtl0cl2axt6CAbR08DW3fsL0-ceC3foWtXfKG97vam_Bv-3cd9-4ZyCDviEnPUu7ozAeSbps3I2-vxb7Ke6kart7L26breuSTztT_8akV-GsZ2i1aJqHfd-4datw9BlFWxBbWi_Nv_I1Mlpv3V9zCSuqGxW9vf0HHgWInQ==)

```mermaid
graph TB
    subgraph "External Clients"
        UI[Web Interface]
        API_CLIENT[API Clients]
    end
    
    subgraph "API Gateway"
        QAPI[queryAPI<br/>Port 8080]
    end
    
    subgraph "Document Processing Pipeline"
        SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
        DI --> DS[docSyncRunner<br/>8083]
        DS --> DC[docCleanRunner<br/>8084]
    end

    subgraph "Synchronization Services"
        CS[clientSyncRunner<br/>8089]
    end
    
    subgraph "Infrastructure Services"
        DB[(PostgreSQL<br/>Database)]
        S3[(AWS S3<br/>Storage)]
        SQS[AWS SQS<br/>Queues]
        COGNITO[AWS Cognito<br/>Authentication]
        PROM[Prometheus<br/>Monitoring]
    end
    
    UI --> QAPI
    API_CLIENT --> QAPI
    
    QAPI --> DB
    QAPI --> S3
    QAPI --> SQS
    QAPI --> COGNITO
    
    SE --> SQS
    DI --> DB
    DS --> DB
    DC --> S3

    CS --> SQS
    
    All_Services --> PROM
```

## Deployment Architecture

### Infrastructure Requirements
- **Compute**: Container orchestration platform (Docker/Kubernetes)
- **Database**: PostgreSQL 17.2+ with connection pooling
- **AWS Services**: S3, SQS, Cognito services
- **Monitoring**: Prometheus-compatible metrics collection
- **Networking**: Load balancer for API endpoints

### Scalability Considerations
- **Horizontal Scaling**: All runners can be scaled independently based on queue depth
- **Database Scaling**: Connection pooling with configurable pool sizes
- **Storage Scaling**: S3 provides virtually unlimited document storage
- **Processing Scaling**: Auto-scaling based on SQS queue metrics

### Security Architecture
- **Authentication**: OAuth2 with AWS Cognito integration and MFA support
- **Authorization**: Enhanced RBAC with Permit.io integration and group-based permissions
- **Data Protection**: Encrypted storage (S3 server-side encryption) with folder path preservation
- **Network Security**: VPC isolation and security groups
- **API Security**: JWT validation and OpenAPI request validation with enhanced token handling

## Integration Points

### External System Dependencies
1. **AWS S3**: Document storage and retrieval
2. **AWS SQS**: Inter-service messaging and event processing
3. **AWS Cognito**: User authentication and session management
4. **PostgreSQL**: Central data persistence

### API Integration
- **REST API**: OpenAPI 3.0.3 specification with Swagger UI
- **Authentication**: Bearer token or OAuth2 flows
- **Rate Limiting**: Dual-layer global and per-IP rate limiting (configurable via environment variables)
- **Content Types**: JSON for API, multipart/form-data for uploads

### Performance Characteristics
- **Throughput**: Designed for high-volume document processing
- **Latency**: Sub-second API response times for most operations
- **Availability**: Distributed architecture supports high availability
- **Durability**: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively