query-orchestration/docs/ai.generated/01-system-overview.md

System Overview

DoczyAI Document Processing Platform

Purpose and Domain

DoczyAI is a cloud-native document processing platform for multi-tenant PDF ingestion and processing. Organizations can upload and organize documents with folders and labels, and store structured field extraction results for downstream analysis.

Core Business Capabilities

• Multi-tenant Document Processing: Isolated processing environments per client
• Batch Document Processing: ZIP archive uploads for processing multiple documents simultaneously
• Automated Processing Pipeline: Seamless flow from document upload to structured output
• Version Management: Complete versioning for configurations
• Export and Integration: Structured data export capabilities for downstream systems
• Enhanced Observability: Comprehensive metrics and monitoring with Prometheus integration

Technology Stack

• Language: Go 1.24.3
• Web Framework: Echo v4 with OpenAPI code generation
• Database: PostgreSQL 17.2 with SQLC for type-safe queries
• Messaging: AWS SQS for event-driven microservices
• Storage: AWS S3 for document storage
• Authentication: AWS Cognito with JWT-based authorization
• Monitoring: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
• Authorization: Permit.io RBAC integration for fine-grained permissions
• Container Platform: Docker with multi-stage builds

Architecture Overview

System Architecture Pattern

The system follows an event-driven microservices architecture with the following characteristics:

• Active services: 1 HTTP API + 5 active queue runners
• Compatibility services: Additional deprecated runners may remain in some deployments
• Asynchronous communication: SQS queues between services with batch processing capabilities
• Background Task Framework: Comprehensive background task execution with periodic scheduling
• State management: PostgreSQL as central data store with enhanced schema for batch operations
• External service integration: AWS managed services for cloud capabilities

High-Level Component Diagram

link to rendered version here

graph TB
    subgraph "External Clients"
        UI[Web Interface]
        API_CLIENT[API Clients]
    end
    
    subgraph "API Gateway"
        QAPI[queryAPI<br/>Port 8080]
    end
    
    subgraph "Document Processing Pipeline"
        SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
        DI --> DS[docSyncRunner<br/>8083]
        DS --> DC[docCleanRunner<br/>8084]
    end

    subgraph "Synchronization Services"
        CS[clientSyncRunner<br/>8089]
    end
    
    subgraph "Infrastructure Services"
        DB[(PostgreSQL<br/>Database)]
        S3[(AWS S3<br/>Storage)]
        SQS[AWS SQS<br/>Queues]
        COGNITO[AWS Cognito<br/>Authentication]
        PROM[Prometheus<br/>Monitoring]
    end
    
    UI --> QAPI
    API_CLIENT --> QAPI
    
    QAPI --> DB
    QAPI --> S3
    QAPI --> SQS
    QAPI --> COGNITO
    
    SE --> SQS
    DI --> DB
    DS --> DB
    DC --> S3

    CS --> SQS
    
    All_Services --> PROM

Deployment Architecture

Infrastructure Requirements

• Compute: Container orchestration platform (Docker/Kubernetes)
• Database: PostgreSQL 17.2+ with connection pooling
• AWS Services: S3, SQS, Cognito services
• Monitoring: Prometheus-compatible metrics collection
• Networking: Load balancer for API endpoints

Scalability Considerations

• Horizontal Scaling: All runners can be scaled independently based on queue depth
• Database Scaling: Connection pooling with configurable pool sizes
• Storage Scaling: S3 provides virtually unlimited document storage
• Processing Scaling: Auto-scaling based on SQS queue metrics

Security Architecture

• Authentication: OAuth2 with AWS Cognito integration and MFA support
• Authorization: Enhanced RBAC with Permit.io integration and group-based permissions
• Multi-Environment Isolation: Multiple customer environments can share a single Cognito user pool. Each server instance is scoped via COGNITO_ENVIRONMENT_ID, and users are tagged with a custom:environment_id attribute. Middleware enforces that users can only access the environment they belong to (super_admin users are exempt).
• Data Protection: Encrypted storage (S3 server-side encryption) with folder path preservation
• Network Security: VPC isolation and security groups
• API Security: JWT validation and OpenAPI request validation with enhanced token handling

Integration Points

External System Dependencies

1. AWS S3: Document storage and retrieval
2. AWS SQS: Inter-service messaging and event processing
3. AWS Cognito: User authentication and session management
4. PostgreSQL: Central data persistence

API Integration

• REST API: OpenAPI 3.0.3 specification with Swagger UI
• Authentication: Bearer token or OAuth2 flows
• Rate Limiting: Dual-layer global and per-IP rate limiting (configurable via environment variables)
• Content Types: JSON for API, multipart/form-data for uploads

Performance Characteristics

• Throughput: Designed for high-volume document processing
• Latency: Sub-second API response times for most operations
• Availability: Distributed architecture supports high availability
• Durability: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively