query-orchestration/docs/ai.generated/README.md

# DoczyAI Query Orchestration Platform Documentation

## Overview

This documentation provides comprehensive coverage of the DoczyAI Query Orchestration Platform - a cloud-native document processing and query orchestration system designed to extract structured information from documents using configurable, hierarchical queries.

## Documentation Structure

### 📋 System Documentation

1. **[System Overview](01-system-overview.md)**
   - Platform purpose and business capabilities
   - Technology stack and architecture overview
   - High-level component diagram
   - Deployment architecture and scalability

2. **[Service Architecture](02-service-architecture.md)**
   - Detailed service catalog (12 services)
   - Document processing pipeline flow
   - Inter-service communication patterns
   - Queue architecture and message formats

3. **[API Documentation](03-api-documentation.md)**
   - REST API endpoints and authentication
   - Request/response schemas and validation
   - OpenAPI specification and code generation
   - Rate limiting and error handling

4. **[Data Architecture](04-data-architecture.md)**
   - Database schema and entity relationships
   - Versioning system and temporal queries
   - Query dependency management
   - Performance optimization strategies

### 🛠️ Developer Documentation

5. **[Getting Started Guide](05-getting-started.md)**
   - Prerequisites and local setup
   - Development environment (Devbox)
   - Common development workflows
   - Testing and debugging guidelines

6. **[Code Organization](06-code-organization.md)**
   - Project structure and package architecture
   - Design patterns and naming conventions
   - Code style guidelines and best practices
   - Testing patterns and examples

7. **[Configuration Management](07-configuration-management.md)**
   - Environment variable hierarchy
   - Service configuration patterns
   - Development vs production settings
   - Security and validation practices

8. **[Operations Guide](08-operations-guide.md)**
   - Deployment procedures
   - Monitoring and alerting setup
   - Log analysis and debugging
   - Backup, recovery, and capacity planning

## Quick Start

### For New Developers
1. Start with **[Getting Started Guide](05-getting-started.md)** for environment setup
2. Read **[Code Organization](06-code-organization.md)** to understand the codebase structure
3. Review **[Configuration Management](07-configuration-management.md)** for local development
4. Explore **[API Documentation](03-api-documentation.md)** for endpoint details

### For System Architects
1. Begin with **[System Overview](01-system-overview.md)** for architectural understanding
2. Study **[Service Architecture](02-service-architecture.md)** for microservices design
3. Review **[Data Architecture](04-data-architecture.md)** for database design
4. Consult **[Operations Guide](08-operations-guide.md)** for deployment planning

### For Operations Teams
1. Focus on **[Operations Guide](08-operations-guide.md)** for deployment and monitoring
2. Reference **[Configuration Management](07-configuration-management.md)** for environment setup
3. Use **[API Documentation](03-api-documentation.md)** for health checks and troubleshooting
4. Understand **[Service Architecture](02-service-architecture.md)** for system dependencies

## System Characteristics

### 🏗️ Architecture
- **Event-driven microservices** with SQS-based communication
- **12 distinct services**: 1 HTTP API + 8 runners + 3 utilities
- **Multi-tenant design** with client isolation
- **Comprehensive versioning** for queries and configurations

### 🔧 Technology Stack
- **Backend**: Go 1.24.0 with Echo framework
- **Database**: PostgreSQL 17.2 with SQLC
- **Cloud**: AWS (S3, SQS, Textract, Cognito)
- **Containers**: Docker with multi-stage builds
- **Monitoring**: Prometheus + OpenTelemetry

### 📊 Key Capabilities
- **Document Processing**: PDF upload, cleaning, and text extraction
- **Query Orchestration**: Configurable data extraction with dependencies
- **Multi-tenant Support**: Isolated client environments
- **Export Operations**: Structured data export in multiple formats
- **Version Management**: Full query and configuration versioning

## Documentation Tools

### PDF Generation

Generate professional PDF documentation with rendered Mermaid diagrams and working internal links:

```bash
# Navigate to the scripts directory
cd docs/ai.generated/scripts

# Run the PDF generation script
./build-and-generate-pdf.sh
```

**Output**: Timestamped PDF file (e.g., `DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf`) in the `docs/ai.generated/` directory.

**Prerequisites:**
- Docker installed and running
- Sufficient disk space (~50MB for build process)

**Features:**
- ✅ **Rendered Mermaid diagrams** - All diagram code blocks converted to PNG graphics
- ✅ **Working internal links** - Cross-references between sections function properly
- ✅ **Professional formatting** - Custom CSS styling with syntax highlighting
- ✅ **Page breaks** - Logical section separation for print/digital viewing
- ✅ **Table of contents** - Auto-generated with clickable navigation

**Technical Details:**
- Uses Docker container with Node.js 18, Chromium, and Mermaid CLI
- Requires `--security-opt seccomp=unconfined --cap-add=SYS_ADMIN` for Mermaid rendering
- Processes all documentation files in logical order with proper formatting
- Generates intermediate PNG files for diagrams (cleaned up automatically)

**Troubleshooting:**
- If Mermaid diagrams don't render, ensure Docker has sufficient privileges
- Large PDFs (>1MB) are normal due to embedded graphics and styling
- Permission errors: Run `chmod +x build-and-generate-pdf.sh`
- Container conflicts: `docker ps -a | grep doczy-docs-generator && docker rm -f doczy-docs-generator`

### Mermaid Diagram Link Generation

For programmatic generation of mermaid.live links in documentation, use the provided scripts:

[link to rendered version here](https://mermaid.live/edit#pako:eNqFkM1KBDEQBl-lyVydycHbCIKieBEU0VtAmqRnE8wfSUcc1n134zqCl8XrR3dR1F7oZEjMIAa4o0gFmQAhUAnozOTdO8HL0z0sJYW-G4e7ggEW50lFjfy7TNsHfMIkqy4uc5XbNnIaW_FTtSqqOMBDgVYJHle2KXZAIc1-PYXLx7Pz09i8_mBvPzBkT52Nuy5H2iZQotOyheebC7gax8trJf4z7H6Nc-MZLHOus5R_Y0gyjoeMb2mm-J3OvG7K4gzEdtlz7gVbCsewhhZsnsXh8AXqY4CE)

```bash
# Generate a mermaid.live URL from a diagram file
cat diagram.mermaid | ./scripts/mermaid-to-url.sh

# Or use Python directly
cat diagram.mermaid | python3 ./scripts/mermaid-to-url.py

# Example usage
echo "graph TD; A-->B" | ./scripts/mermaid-to-url.sh
# Output: https://mermaid.live/edit#pako:encoded_diagram
```

**Features:**
- **Pako compression**: Uses proper deflate compression for URL optimization
- **URL-safe encoding**: Compatible with mermaid.live format
- **Pipe-friendly**: Accepts mermaid syntax via stdin
- **Cross-platform**: Works with bash or Python 3

**Use cases:**
- Automated documentation generation
- CI/CD pipeline diagram embedding
- Dynamic diagram URL creation for wikis/docs

### Post-Processing Documentation with Mermaid Links

After generating documentation, you can automatically add rendered diagram links to all mermaid diagrams:

```bash
# Add mermaid.live links to all markdown files in a directory
python3 ./scripts/add-mermaid-links.py --target-dir ./docs

# Preview changes without modifying files
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --dry-run

# Process subdirectories recursively
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive
```

**Features:**
- **Automatic detection**: Finds mermaid diagrams by language tag or content keywords
- **Non-destructive**: Preserves all original content, only adds links
- **Duplicate prevention**: Skips diagrams that already have links
- **Flexible targeting**: Process single directories or recursive scans

**Workflow integration:**
1. Generate your documentation with embedded mermaid diagrams
2. Run the link addition script as a post-processing step
3. All mermaid diagrams now include clickable links to rendered versions

### Complete Documentation Workflow

For comprehensive documentation generation and maintenance:

```bash
# 1. Navigate to the ai.generated directory
cd docs/ai.generated

# 2. Generate updated PDF with rendered diagrams
cd scripts && ./build-and-generate-pdf.sh && cd ..

# 3. (Optional) Add mermaid.live links to markdown files
python3 ./scripts/add-mermaid-links.py --target-dir . --recursive

# 4. Verify output
ls -la DoczyAI-Documentation-*.pdf
```

This workflow ensures all documentation formats are current with proper diagram rendering and cross-references.

## Contributing

### Development Workflow
1. **Setup**: Follow the Getting Started Guide
2. **Development**: Use task-based workflows (`task --list`)
3. **Testing**: Maintain 80% total, 60% function coverage
4. **Documentation**: Update relevant docs with changes

### Code Standards
- **Language**: Go with standard project layout
- **Testing**: Table-driven tests with testcontainers
- **API**: Contract-first with OpenAPI generation
- **Database**: Migration-based schema evolution

## Support and Resources

### Internal Resources
- **Swagger UI**: http://localhost:8080/swagger/ (when running locally)
- **Metrics**: http://localhost:9091 (Prometheus)
- **Health Checks**: http://localhost:8080/health

### External Dependencies
- **AWS Services**: S3, SQS, Textract, Cognito
- **Database**: PostgreSQL 17.2+
- **Monitoring**: Prometheus-compatible metrics

---

*This documentation is automatically generated and maintained alongside the codebase. For questions or improvements, please contact the platform team.*