Files

T

Jay Brown 0ddae4f91e Merged in feature/remove-query (pull request #201 )

remove query from codebase part 1

* remove query

* fix localstack run

2026-01-14 17:59:04 +00:00

6.1 KiB

Raw Blame History

System Overview

DoczyAI Document Processing Platform

Purpose and Domain

DoczyAI is a cloud-native document processing platform designed to extract structured information from documents (primarily PDFs). The system serves as a multi-tenant platform where organizations can upload documents, perform text extraction, and organize documents with folders and labels for business intelligence and analysis.

Core Business Capabilities

Multi-tenant Document Processing: Isolated processing environments per client
Batch Document Processing: ZIP archive uploads for processing multiple documents simultaneously
Text Extraction Pipeline: Automated OCR and text extraction from documents
Automated Processing Pipeline: Seamless flow from document upload to structured output
Version Management: Complete versioning for configurations
Export and Integration: Structured data export capabilities for downstream systems
Enhanced Observability: Comprehensive metrics and monitoring with Prometheus integration

Technology Stack

Language: Go 1.24.3
Web Framework: Echo v4 with OpenAPI code generation
Database: PostgreSQL 17.2 with SQLC for type-safe queries
Messaging: AWS SQS for event-driven microservices
Storage: AWS S3 for document storage
Text Processing: AWS Textract for OCR and text extraction
Authentication: AWS Cognito with JWT-based authorization
Monitoring: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
Authorization: Permit.io RBAC integration for fine-grained permissions
Container Platform: Docker with multi-stage builds

Architecture Overview

System Architecture Pattern

The system follows an event-driven microservices architecture with the following characteristics:

8 distinct services: 1 HTTP API + 6 queue-based runners + 1 utility service
Asynchronous communication: SQS queues between services with batch processing capabilities
Background Task Framework: Comprehensive background task execution with periodic scheduling
State management: PostgreSQL as central data store with enhanced schema for batch operations
External service integration: AWS managed services for cloud capabilities

High-Level Component Diagram

link to rendered version here

graph TB
    subgraph "External Clients"
        UI[Web Interface]
        API_CLIENT[API Clients]
    end
    
    subgraph "API Gateway"
        QAPI[queryAPI<br/>Port 8080]
    end
    
    subgraph "Document Processing Pipeline"
        SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
        DI --> DS[docSyncRunner<br/>8083]
        DS --> DC[docCleanRunner<br/>8084]
        DC --> DT[docTextRunner<br/>8085]
    end

    subgraph "Synchronization Services"
        CS[clientSyncRunner<br/>8089]
    end
    
    subgraph "Infrastructure Services"
        DB[(PostgreSQL<br/>Database)]
        S3[(AWS S3<br/>Storage)]
        SQS[AWS SQS<br/>Queues]
        TEXTRACT[AWS Textract<br/>Text Extraction]
        COGNITO[AWS Cognito<br/>Authentication]
        PROM[Prometheus<br/>Monitoring]
    end
    
    UI --> QAPI
    API_CLIENT --> QAPI
    
    QAPI --> DB
    QAPI --> S3
    QAPI --> SQS
    QAPI --> COGNITO
    
    SE --> SQS
    DI --> DB
    DS --> DB
    DC --> S3
    DT --> TEXTRACT

    CS --> SQS
    
    All_Services --> PROM

Deployment Architecture

Infrastructure Requirements

Compute: Container orchestration platform (Docker/Kubernetes)
Database: PostgreSQL 17.2+ with connection pooling
AWS Services: S3, SQS, Textract, Cognito services
Monitoring: Prometheus-compatible metrics collection
Networking: Load balancer for API endpoints

Scalability Considerations

Horizontal Scaling: All runners can be scaled independently based on queue depth
Database Scaling: Connection pooling with configurable pool sizes
Storage Scaling: S3 provides virtually unlimited document storage
Processing Scaling: Auto-scaling based on SQS queue metrics

Security Architecture

Authentication: OAuth2 with AWS Cognito integration and MFA support
Authorization: Enhanced RBAC with Permit.io integration and group-based permissions
Data Protection: Encrypted storage (S3 server-side encryption) with folder path preservation
Network Security: VPC isolation and security groups
API Security: JWT validation and OpenAPI request validation with enhanced token handling

Integration Points

External System Dependencies

AWS S3: Document storage and retrieval
AWS SQS: Inter-service messaging and event processing
AWS Textract: OCR and text extraction from documents
AWS Cognito: User authentication and session management
PostgreSQL: Central data persistence

API Integration

REST API: OpenAPI 3.0.3 specification with Swagger UI
Authentication: Bearer token or OAuth2 flows
Rate Limiting: 100 requests per 60-second window
Content Types: JSON for API, multipart/form-data for uploads

Performance Characteristics

Throughput: Designed for high-volume document processing
Latency: Sub-second API response times for most operations
Availability: Distributed architecture supports high availability
Durability: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively

6.1 KiB Raw Blame History