query-orchestration/docs/ai.generated

DoczyAI Document Processing Platform Documentation

Overview

This documentation provides comprehensive coverage of the DoczyAI Document Processing Platform - a cloud-native document processing system designed to extract structured information from documents.

Architecture Diagram

The diagram below shows the full API layer architecture — from HTTP endpoints through middleware, controllers, services, and the repository layer, ending at the database and external services. Request flow runs left to right.

View scalable SVG version

Documentation Structure

📋 System Documentation

1. System Overview

   • Platform purpose and business capabilities
   • Technology stack and architecture overview
   • High-level component diagram
   • Deployment architecture and scalability

2. Service Architecture

   • Detailed service catalog (active + compatibility services)
   • Document processing pipeline flow
   • Inter-service communication patterns
   • Queue architecture and message formats

3. API Documentation

   • REST API endpoints and authentication
   • Request/response schemas and validation
   • OpenAPI specification and code generation
   • Rate limiting and error handling

4. Data Architecture

   • Database schema and entity relationships
   • Versioning system
   • Document organization (folders and labels)
   • Performance optimization strategies

🛠️ Developer Documentation

5. Getting Started Guide

   • Prerequisites and local setup
   • Development environment (Devbox)
   • Common development workflows
   • Testing and debugging guidelines

6. Code Organization

   • Project structure and package architecture
   • Design patterns and naming conventions
   • Code style guidelines and best practices
   • Testing patterns and examples

7. Configuration Management

   • Environment variable hierarchy
   • Service configuration patterns
   • Development vs production settings
   • Security and validation practices

8. Operations Guide

   • Deployment procedures
   • Monitoring and alerting setup
   • Log analysis and debugging
   • Backup, recovery, and capacity planning

9. Authentication Guide

   • JWT authentication with AWS Cognito
   • Token types and refresh strategy
   • CORS configuration

10. Appendix

    • S3 Storage Guide: BucketKey path format, segment reference, partitioning
    • Watching S3 for client document imports
    • Document recovery from S3 buckets

11. Custom Metadata Guide

    • Overview: custom schemas vs legacy field extractions
    • Authorization matrix (super_admin, user_admin, client_user, auditor)
    • Platform operator workflow: define schema, assign to document
    • End-user workflow: write, read current, read by version, read history
    • Schema definition requirements and sample schemas
    • Schema versioning: how POST .../versions works, status lifecycle
    • Reset-metadata runbook: schema upgrade flow with metadata wipe warning
    • Cascade deletion behavior (no special steps needed)
    • Permit.io deployment note

Quick Start

For New Developers

1. Start with Getting Started Guide for environment setup
2. Read Code Organization to understand the codebase structure
3. Review Configuration Management for local development
4. Explore API Documentation for endpoint details

For System Architects

1. Begin with System Overview for architectural understanding
2. Study Service Architecture for microservices design
3. Review Data Architecture for database design
4. Consult Operations Guide for deployment planning

For Operations Teams

1. Focus on Operations Guide for deployment and monitoring
2. Reference Configuration Management for environment setup
3. Use API Documentation for health checks and troubleshooting
4. Understand Service Architecture for system dependencies

System Characteristics

🏗️ Architecture

• Event-driven microservices with SQS-based communication
• Active services: 1 HTTP API + 5 active runners
• Compatibility services: additional deprecated runner binaries may still be present in some deployments
• Multi-tenant design with client isolation
• Comprehensive versioning for configurations

🔧 Technology Stack

• Backend: Go 1.24.0 with Echo framework
• Database: PostgreSQL 17.2 with SQLC
• Cloud: AWS (S3, SQS, Cognito)
• Containers: Docker with multi-stage builds
• Monitoring: Prometheus + OpenTelemetry

📊 Key Capabilities

• Document Processing: Multi-format document upload (PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML), download, initialization, sync-gating, and cleaning
• Multi-tenant Support: Isolated client environments
• Document Organization: Folders and labels for document management
• Export Operations: Structured data export in multiple formats

Documentation Tools

PDF Generation

Generate professional PDF documentation with rendered Mermaid diagrams and working internal links:

# Navigate to the scripts directory
cd docs/ai.generated/scripts

# Run the PDF generation script
./build-and-generate-pdf.sh

Output: Timestamped PDF file (e.g., DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf) in the docs/ai.generated/ directory.

Prerequisites:

• Docker installed and running
• Sufficient disk space (~50MB for build process)

Features:

• ✅ Rendered Mermaid diagrams - All diagram code blocks converted to PNG graphics
• ✅ Working internal links - Cross-references between sections function properly
• ✅ Professional formatting - Custom CSS styling with syntax highlighting
• ✅ Page breaks - Logical section separation for print/digital viewing
• ✅ Table of contents - Auto-generated with clickable navigation

Technical Details:

• Uses Docker container with Node.js 18, Chromium, and Mermaid CLI
• Requires --security-opt seccomp=unconfined --cap-add=SYS_ADMIN for Mermaid rendering
• Processes all documentation files in logical order with proper formatting
• Generates intermediate PNG files for diagrams (cleaned up automatically)

Troubleshooting:

• If Mermaid diagrams don't render, ensure Docker has sufficient privileges
• Large PDFs (>1MB) are normal due to embedded graphics and styling
• Permission errors: Run chmod +x build-and-generate-pdf.sh
• Container conflicts: docker ps -a | grep doczy-docs-generator && docker rm -f doczy-docs-generator

Mermaid Diagram Link Generation

For programmatic generation of mermaid.live links in documentation, use the provided scripts:

link to rendered version here

# Generate a mermaid.live URL from a diagram file
cat diagram.mermaid | ./scripts/mermaid-to-url.sh

# Or use Python directly
cat diagram.mermaid | python3 ./scripts/mermaid-to-url.py

# Example usage
echo "graph TD; A-->B" | ./scripts/mermaid-to-url.sh
# Output: https://mermaid.live/edit#pako:encoded_diagram

Features:

• Pako compression: Uses proper deflate compression for URL optimization
• URL-safe encoding: Compatible with mermaid.live format
• Pipe-friendly: Accepts mermaid syntax via stdin
• Cross-platform: Works with bash or Python 3

Use cases:

• Automated documentation generation
• CI/CD pipeline diagram embedding
• Dynamic diagram URL creation for wikis/docs

Post-Processing Documentation with Mermaid Links

After generating documentation, you can automatically add rendered diagram links to all mermaid diagrams:

# Add mermaid.live links to all markdown files in a directory
python3 ./scripts/add-mermaid-links.py --target-dir ./docs

# Preview changes without modifying files
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --dry-run

# Process subdirectories recursively
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive

Features:

• Automatic detection: Finds mermaid diagrams by language tag or content keywords
• Non-destructive: Preserves all original content, only adds links
• Duplicate prevention: Skips diagrams that already have links
• Flexible targeting: Process single directories or recursive scans

Workflow integration:

1. Generate your documentation with embedded mermaid diagrams
2. Run the link addition script as a post-processing step
3. All mermaid diagrams now include clickable links to rendered versions

Complete Documentation Workflow

For comprehensive documentation generation and maintenance:

# 1. Navigate to the ai.generated directory
cd docs/ai.generated

# 2. Generate updated PDF with rendered diagrams
cd scripts && ./build-and-generate-pdf.sh && cd ..

# 3. (Optional) Add mermaid.live links to markdown files
python3 ./scripts/add-mermaid-links.py --target-dir . --recursive

# 4. Verify output
ls -la DoczyAI-Documentation-*.pdf

This workflow ensures all documentation formats are current with proper diagram rendering and cross-references.

Contributing

Development Workflow

1. Setup: Follow the Getting Started Guide
2. Development: Use task-based workflows (task --list)
3. Testing: Maintain 80% total, 60% function coverage
4. Documentation: Update relevant docs with changes

Code Standards

• Language: Go with standard project layout
• Testing: Table-driven tests with testcontainers
• API: Contract-first with OpenAPI generation
• Database: Migration-based schema evolution

Support and Resources

Internal Resources

• Swagger UI: http://localhost:8080/swagger/ (when running locally)
• Metrics: http://localhost:9091 (Prometheus)
• Health Checks: http://localhost:8080/health

External Dependencies

• AWS Services: S3, SQS, Cognito
• Database: PostgreSQL 17.2+
• Monitoring: Prometheus-compatible metrics

---

This documentation is automatically generated and maintained alongside the codebase. For questions or improvements, please contact the platform team.