query-orchestration/docs/ai.generated/01-system-overview.md

# System Overview

## DoczyAI Document Processing Platform

### Purpose and Domain
DoczyAI is a cloud-native document processing platform for multi-tenant document ingestion and processing. The system supports a wide range of file types including PDF, images (TIFF, JPEG, PNG, BMP), Office documents (DOCX, XLSX, PPTX), plain text, and email (EML). Organizations can upload and organize documents with folders and labels, and store structured field extraction results for downstream analysis.

### Core Business Capabilities
- **Multi-tenant Document Processing**: Isolated processing environments per client
- **Batch Document Processing**: ZIP archive uploads for processing multiple documents simultaneously
- **Automated Processing Pipeline**: Seamless flow from document upload to structured output
- **Version Management**: Complete versioning for configurations
- **Export and Integration**: Structured data export capabilities for downstream systems
- **Enhanced Observability**: Comprehensive metrics and monitoring with Prometheus integration

### Technology Stack
- **Language**: Go 1.24.3
- **Web Framework**: Echo v4 with OpenAPI code generation
- **Database**: PostgreSQL 17.2 with SQLC for type-safe queries
- **Messaging**: AWS SQS for event-driven microservices
- **Storage**: AWS S3 for document storage
- **Authentication**: AWS Cognito with JWT-based authorization
- **Monitoring**: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
- **Authorization**: Permit.io RBAC integration for fine-grained permissions
- **Container Platform**: Docker with multi-stage builds

## Architecture Overview

### System Architecture Pattern
The system follows an **event-driven microservices architecture** with the following characteristics:
- **Active services**: 1 HTTP API + 5 active queue runners
- **Compatibility services**: Additional deprecated runners may remain in some deployments
- **Asynchronous communication**: SQS queues between services with batch processing capabilities
- **Background Task Framework**: Comprehensive background task execution with periodic scheduling
- **State management**: PostgreSQL as central data store with enhanced schema for batch operations
- **External service integration**: AWS managed services for cloud capabilities

### High-Level Component Diagram

[link to rendered version here](https://mermaid.live/edit#pako:eNqFlGtr2zAUhv-K8KcOVtYtG0vHKCSyKYa2sSM3HTilKM5JYnCkTpe2Wel_ny4OlZ1C_MGWjp5z0asjv0YVX0L0C0VrQR83qBjPGTKP1AtvmEfJiwLBaINwUwNTch55xD63aXkHC5Qyg6xoBffvS6MsfcBXaXJTlGa4d24BYEs_OMhm2Uuq4JnuwkS5sZd_NYidGfxeiC8XGRcKDc-GZ0dDxrzSW5McZYJXIGXN1iirH6GpGYQ5SFJKxQUkTwaeasZAuEwmydd7dHp6geK0XPIqZXVv-Vuw7zj1KLEo2bGqiw5ClHgUWxQ3QFmX_R6y2LOFZQt46VXwI0QLh-bEC3ZYw88Azn0N-dTDXXB4VFsbfCM4q_9RVXOGCIin2ogc6opJWbnTP6zkPKxk1hY8AyFNrAP6_PhRp2wlqFRCV0oL-LCaeFyeZFyqtQCSX7nQMVV0QSV8Cqohg_JkdEfM1yHENAZddwkjsCNy4pBcgwYZAEXyp5iOcOEoe2SCVsqhdoISbzA7DXzw5PImLSbOBfO16TTuPEZabYyCdUV7Dtl0cl2axt6CAbR08DW3fsL0-ceC3foWtXfKG97vam_Bv-3cd9-4ZyCDviEnPUu7ozAeSbps3I2-vxb7Ke6kart7L26breuSTztT_8akV-GsZ2i1aJqHfd-4datw9BlFWxBbWi_Nv_I1Mlpv3V9zCSuqGxW9vf0HHgWInQ==)

```mermaid
graph TB
    subgraph "External Clients"
        UI[Web Interface]
        API_CLIENT[API Clients]
    end
    
    subgraph "API Gateway"
        QAPI[queryAPI<br/>Port 8080]
    end
    
    subgraph "Document Processing Pipeline"
        SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
        DI --> DS[docSyncRunner<br/>8083]
        DS --> DC[docCleanRunner<br/>8084]
    end

    subgraph "Synchronization Services"
        CS[clientSyncRunner<br/>8089]
    end
    
    subgraph "Infrastructure Services"
        DB[(PostgreSQL<br/>Database)]
        S3[(AWS S3<br/>Storage)]
        SQS[AWS SQS<br/>Queues]
        COGNITO[AWS Cognito<br/>Authentication]
        PROM[Prometheus<br/>Monitoring]
    end
    
    UI --> QAPI
    API_CLIENT --> QAPI
    
    QAPI --> DB
    QAPI --> S3
    QAPI --> SQS
    QAPI --> COGNITO
    
    SE --> SQS
    DI --> DB
    DS --> DB
    DC --> S3

    CS --> SQS
    
    All_Services --> PROM
```

### API Layer Architecture

The diagram below shows the internal layered architecture of the queryAPI service — how an inbound HTTP request passes through middleware, controllers, and services before reaching the database.

[![Query Orchestration API — Software Architecture Layers](architecture-diagram.png)](architecture-diagram.svg)

*[View scalable SVG version](architecture-diagram.svg)*

## Deployment Architecture

### Infrastructure Requirements
- **Compute**: Container orchestration platform (Docker/Kubernetes)
- **Database**: PostgreSQL 17.2+ with connection pooling
- **AWS Services**: S3, SQS, Cognito services
- **Monitoring**: Prometheus-compatible metrics collection
- **Networking**: Load balancer for API endpoints

### Scalability Considerations
- **Horizontal Scaling**: All runners can be scaled independently based on queue depth
- **Database Scaling**: Connection pooling with configurable pool sizes
- **Storage Scaling**: S3 provides virtually unlimited document storage
- **Processing Scaling**: Auto-scaling based on SQS queue metrics

### Security Architecture
- **Authentication**: OAuth2 with AWS Cognito integration and MFA support
- **Authorization**: Enhanced RBAC with Permit.io integration and group-based permissions
- **Multi-Environment Isolation**: Multiple customer environments can share a single Cognito user pool. Each server instance is scoped via `COGNITO_ENVIRONMENT_ID`, and users are tagged with a `custom:environment_id` attribute. Middleware enforces that users can only access the environment they belong to (super_admin users are exempt).
- **Data Protection**: Encrypted storage (S3 server-side encryption) with folder path preservation
- **Network Security**: VPC isolation and security groups
- **API Security**: JWT validation and OpenAPI request validation with enhanced token handling

## Integration Points

### External System Dependencies
1. **AWS S3**: Document storage and retrieval
2. **AWS SQS**: Inter-service messaging and event processing
3. **AWS Cognito**: User authentication and session management
4. **PostgreSQL**: Central data persistence

### API Integration
- **REST API**: OpenAPI 3.0.3 specification with Swagger UI
- **Authentication**: Bearer token or OAuth2 flows
- **Rate Limiting**: Dual-layer global and per-IP rate limiting (configurable via environment variables)
- **Content Types**: JSON for API, multipart/form-data for uploads

### Performance Characteristics
- **Throughput**: Designed for high-volume document processing
- **Latency**: Sub-second API response times for most operations
- **Availability**: Distributed architecture supports high availability
- **Durability**: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively