query-orchestration/docs/ai.generated/01-system-overview.md

System Overview

DoczyAI Query Orchestration Platform

Purpose and Domain

DoczyAI is a cloud-native document processing and query orchestration platform designed to extract structured information from documents (primarily PDFs) using configurable, hierarchical queries. The system serves as a multi-tenant platform where organizations can upload documents, define custom data extraction rules, and receive structured results for business intelligence and analysis.

Core Business Capabilities

• Multi-tenant Document Processing: Isolated processing environments per client
• Batch Document Processing: ZIP archive uploads for processing multiple documents simultaneously
• Configurable Query System: Custom data extraction rules with dependency management
• Automated Processing Pipeline: Seamless flow from document upload to structured output
• Version Management: Complete versioning for queries and configurations
• Export and Integration: Structured data export capabilities for downstream systems
• Enhanced Observability: Comprehensive metrics and monitoring with Prometheus integration

Technology Stack

• Language: Go 1.24.3
• Web Framework: Echo v4 with OpenAPI code generation
• Database: PostgreSQL 17.2 with SQLC for type-safe queries
• Messaging: AWS SQS for event-driven microservices
• Storage: AWS S3 for document storage
• Text Processing: AWS Textract for OCR and text extraction
• Authentication: AWS Cognito with JWT-based authorization
• Monitoring: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
• Authorization: Permit.io RBAC integration for fine-grained permissions
• Container Platform: Docker with multi-stage builds

Architecture Overview

System Architecture Pattern

The system follows an event-driven microservices architecture with the following characteristics:

• 12 distinct services: 1 HTTP API + 8 queue-based runners + 3 utility services
• Asynchronous communication: SQS queues between services with batch processing capabilities
• Background Task Framework: Comprehensive background task execution with periodic scheduling
• State management: PostgreSQL as central data store with enhanced schema for batch operations
• External service integration: AWS managed services for cloud capabilities

High-Level Component Diagram

link to rendered version here

graph TB
    subgraph "External Clients"
        UI[Web Interface]
        API_CLIENT[API Clients]
    end
    
    subgraph "API Gateway"
        QAPI[queryAPI<br/>Port 8080]
    end
    
    subgraph "Document Processing Pipeline"
        SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
        DI --> DS[docSyncRunner<br/>8083]
        DS --> DC[docCleanRunner<br/>8084]
        DC --> DT[docTextRunner<br/>8085]
        DT --> QS[querySyncRunner<br/>8087]
        QS --> QR[queryRunner<br/>8088]
    end
    
    subgraph "Synchronization Services"
        CS[clientSyncRunner<br/>8089]
        QVS[queryVersionSyncRunner<br/>8090]
    end
    
    subgraph "Infrastructure Services"
        DB[(PostgreSQL<br/>Database)]
        S3[(AWS S3<br/>Storage)]
        SQS[AWS SQS<br/>Queues]
        TEXTRACT[AWS Textract<br/>Text Extraction]
        COGNITO[AWS Cognito<br/>Authentication]
        PROM[Prometheus<br/>Monitoring]
    end
    
    UI --> QAPI
    API_CLIENT --> QAPI
    
    QAPI --> DB
    QAPI --> S3
    QAPI --> SQS
    QAPI --> COGNITO
    
    SE --> SQS
    DI --> DB
    DS --> DB
    DC --> S3
    DT --> TEXTRACT
    QS --> DB
    QR --> DB
    
    CS --> SQS
    QVS --> SQS
    
    All_Services --> PROM

Deployment Architecture

Infrastructure Requirements

• Compute: Container orchestration platform (Docker/Kubernetes)
• Database: PostgreSQL 17.2+ with connection pooling
• AWS Services: S3, SQS, Textract, Cognito services
• Monitoring: Prometheus-compatible metrics collection
• Networking: Load balancer for API endpoints

Scalability Considerations

• Horizontal Scaling: All runners can be scaled independently based on queue depth
• Database Scaling: Connection pooling with configurable pool sizes
• Storage Scaling: S3 provides virtually unlimited document storage
• Processing Scaling: Auto-scaling based on SQS queue metrics

Security Architecture

• Authentication: OAuth2 with AWS Cognito integration and MFA support
• Authorization: Enhanced RBAC with Permit.io integration and group-based permissions
• Data Protection: Encrypted storage (S3 server-side encryption) with folder path preservation
• Network Security: VPC isolation and security groups
• API Security: JWT validation and OpenAPI request validation with enhanced token handling

Integration Points

External System Dependencies

1. AWS S3: Document storage and retrieval
2. AWS SQS: Inter-service messaging and event processing
3. AWS Textract: OCR and text extraction from documents
4. AWS Cognito: User authentication and session management
5. PostgreSQL: Central data persistence and query processing

API Integration

• REST API: OpenAPI 3.0.3 specification with Swagger UI
• Authentication: Bearer token or OAuth2 flows
• Rate Limiting: 100 requests per 60-second window
• Content Types: JSON for API, multipart/form-data for uploads

Performance Characteristics

• Throughput: Designed for high-volume document processing
• Latency: Sub-second API response times for most operations
• Availability: Distributed architecture supports high availability
• Durability: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively