Files

T

Jay Brown 17fc813823 Merged in feature/mutable-metadata1 (pull request #221 )

M1, M2 and M3 complete

* M1, M2 and M3 complete

* review changes

* docs

* docs

2026-04-16 23:11:26 +00:00

6.7 KiB

Raw Permalink Blame History

System Overview

DoczyAI Document Processing Platform

Purpose and Domain

DoczyAI is a cloud-native document processing platform for multi-tenant document ingestion and processing. The system supports a wide range of file types including PDF, images (TIFF, JPEG, PNG, BMP), Office documents (DOCX, XLSX, PPTX), plain text, and email (EML). Organizations can upload and organize documents with folders and labels, and store structured field extraction results for downstream analysis.

Core Business Capabilities

Multi-tenant Document Processing: Isolated processing environments per client
Batch Document Processing: ZIP archive uploads for processing multiple documents simultaneously
Automated Processing Pipeline: Seamless flow from document upload to structured output
Version Management: Complete versioning for configurations
Export and Integration: Structured data export capabilities for downstream systems
Enhanced Observability: Comprehensive metrics and monitoring with Prometheus integration

Technology Stack

Language: Go 1.24.3
Web Framework: Echo v4 with OpenAPI code generation
Database: PostgreSQL 17.2 with SQLC for type-safe queries
Messaging: AWS SQS for event-driven microservices
Storage: AWS S3 for document storage
Authentication: AWS Cognito with JWT-based authorization
Monitoring: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
Authorization: Permit.io RBAC integration for fine-grained permissions
Container Platform: Docker with multi-stage builds

Architecture Overview

System Architecture Pattern

The system follows an event-driven microservices architecture with the following characteristics:

Active services: 1 HTTP API + 5 active queue runners
Compatibility services: Additional deprecated runners may remain in some deployments
Asynchronous communication: SQS queues between services with batch processing capabilities
Background Task Framework: Comprehensive background task execution with periodic scheduling
State management: PostgreSQL as central data store with enhanced schema for batch operations
External service integration: AWS managed services for cloud capabilities

High-Level Component Diagram

link to rendered version here

graph TB
    subgraph "External Clients"
        UI[Web Interface]
        API_CLIENT[API Clients]
    end
    
    subgraph "API Gateway"
        QAPI[queryAPI<br/>Port 8080]
    end
    
    subgraph "Document Processing Pipeline"
        SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
        DI --> DS[docSyncRunner<br/>8083]
        DS --> DC[docCleanRunner<br/>8084]
    end

    subgraph "Synchronization Services"
        CS[clientSyncRunner<br/>8089]
    end
    
    subgraph "Infrastructure Services"
        DB[(PostgreSQL<br/>Database)]
        S3[(AWS S3<br/>Storage)]
        SQS[AWS SQS<br/>Queues]
        COGNITO[AWS Cognito<br/>Authentication]
        PROM[Prometheus<br/>Monitoring]
    end
    
    UI --> QAPI
    API_CLIENT --> QAPI
    
    QAPI --> DB
    QAPI --> S3
    QAPI --> SQS
    QAPI --> COGNITO
    
    SE --> SQS
    DI --> DB
    DS --> DB
    DC --> S3

    CS --> SQS
    
    All_Services --> PROM

API Layer Architecture

The diagram below shows the internal layered architecture of the queryAPI service — how an inbound HTTP request passes through middleware, controllers, and services before reaching the database.

View scalable SVG version

Deployment Architecture

Infrastructure Requirements

Compute: Container orchestration platform (Docker/Kubernetes)
Database: PostgreSQL 17.2+ with connection pooling
AWS Services: S3, SQS, Cognito services
Monitoring: Prometheus-compatible metrics collection
Networking: Load balancer for API endpoints

Scalability Considerations

Horizontal Scaling: All runners can be scaled independently based on queue depth
Database Scaling: Connection pooling with configurable pool sizes
Storage Scaling: S3 provides virtually unlimited document storage
Processing Scaling: Auto-scaling based on SQS queue metrics

Security Architecture

Authentication: OAuth2 with AWS Cognito integration and MFA support
Authorization: Enhanced RBAC with Permit.io integration and group-based permissions
Multi-Environment Isolation: Multiple customer environments can share a single Cognito user pool. Each server instance is scoped via COGNITO_ENVIRONMENT_ID, and users are tagged with a custom:environment_id attribute. Middleware enforces that users can only access the environment they belong to (super_admin users are exempt).
Data Protection: Encrypted storage (S3 server-side encryption) with folder path preservation
Network Security: VPC isolation and security groups
API Security: JWT validation and OpenAPI request validation with enhanced token handling

Integration Points

External System Dependencies

AWS S3: Document storage and retrieval
AWS SQS: Inter-service messaging and event processing
AWS Cognito: User authentication and session management
PostgreSQL: Central data persistence

API Integration

REST API: OpenAPI 3.0.3 specification with Swagger UI
Authentication: Bearer token or OAuth2 flows
Rate Limiting: Dual-layer global and per-IP rate limiting (configurable via environment variables)
Content Types: JSON for API, multipart/form-data for uploads

Performance Characteristics

Throughput: Designed for high-volume document processing
Latency: Sub-second API response times for most operations
Availability: Distributed architecture supports high availability
Durability: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively

6.7 KiB Raw Permalink Blame History