# DoczyAI Document Processing Platform Documentation

## Overview

This documentation provides comprehensive coverage of the DoczyAI Document Processing Platform - a cloud-native document processing system designed to extract structured information from documents.

## Architecture Diagram

The diagram below shows the full API layer architecture — from HTTP endpoints through middleware, controllers, services, and the repository layer, ending at the database and external services. Request flow runs left to right.

[![Query Orchestration API — Software Architecture Layers](architecture-diagram.png)](architecture-diagram.svg)

*[View scalable SVG version](architecture-diagram.svg)*

## Documentation Structure

### 📋 System Documentation

1. **[System Overview](01-system-overview.md)**
   - Platform purpose and business capabilities
   - Technology stack and architecture overview
   - High-level component diagram
   - Deployment architecture and scalability

2. **[Service Architecture](02-service-architecture.md)**
   - Detailed service catalog (active + compatibility services)
   - Document processing pipeline flow
   - Inter-service communication patterns
   - Queue architecture and message formats

3. **[API Documentation](03-api-documentation.md)**
   - REST API endpoints and authentication
   - Request/response schemas and validation
   - OpenAPI specification and code generation
   - Rate limiting and error handling

4. **[Data Architecture](04-data-architecture.md)**
   - Database schema and entity relationships
   - Versioning system
   - Document organization (folders and labels)
   - Performance optimization strategies

### 🛠️ Developer Documentation

5. **[Getting Started Guide](05-getting-started.md)**
   - Prerequisites and local setup
   - Development environment (Devbox)
   - Common development workflows
   - Testing and debugging guidelines

6. **[Code Organization](06-code-organization.md)**
   - Project structure and package architecture
   - Design patterns and naming conventions
   - Code style guidelines and best practices
   - Testing patterns and examples

7. **[Configuration Management](07-configuration-management.md)**
   - Environment variable hierarchy
   - Service configuration patterns
   - Development vs production settings
   - Security and validation practices

8. **[Operations Guide](08-operations-guide.md)**
   - Deployment procedures
   - Monitoring and alerting setup
   - Log analysis and debugging
   - Backup, recovery, and capacity planning

9. **[Authentication Guide](09-authentication-guide.md)**
   - JWT authentication with AWS Cognito
   - Token types and refresh strategy
   - CORS configuration

10. **[Appendix](10-appendix.md)**
    - S3 Storage Guide: BucketKey path format, segment reference, partitioning
    - Watching S3 for client document imports
    - Document recovery from S3 buckets

11. **[Custom Metadata Guide](11-custom-metadata-guide.md)**
    - Overview: custom schemas vs legacy field extractions
    - Authorization matrix (super_admin, user_admin, client_user, auditor)
    - Platform operator workflow: define schema, assign to document
    - End-user workflow: write, read current, read by version, read history
    - Schema definition requirements and sample schemas
    - Schema versioning: how POST .../versions works, status lifecycle
    - Reset-metadata runbook: schema upgrade flow with metadata wipe warning
    - Cascade deletion behavior (no special steps needed)
    - Permit.io deployment note

**[File Validity Checks](file.filters.md)**
   - PDF-specific validation (signature, structure, per-page size, dimensions, DPI) and SHA-256 deduplication
   - Per-format checks for DOCX, XLSX, PPTX, TIFF, JPEG, PNG, BMP, TXT, EML
   - Common pre-pipeline checks (extension allowlist, ZIP-archive limits, OLE2-encryption sniff)
   - Failure-reason catalog (`InvalidDocumentReason`)

12. **[Chatbot Guide](12-chatbot-guide.md)**
    - Overview: queryAPI's facade over the Aaria FastAPI agent service
    - Architecture: `bot_*` data model, AddTurn tx1+FastAPI+tx2 flow, state stripping
    - Configuration reference: 5 `CHATBOT_*` env vars, derived values, validation
    - API reference: 8 user routes + 3 super-admin routes; HTTP status mapping
    - Operator runbook: failed-turn classifications (`errored`, `abandoned`, `session_deleted`) and recovery paths
    - Rate-limit overrides: M6 behavior change and Echo `:param` template form
    - Known warts and tradeoffs (CHECK guard, error chaining, label-record divergence, etc.)

13. **[Two-Pool Service Accounts](13-two-pool-service-accounts.md)**
    - Multi-issuer JWT validation (pool A + pool B) and `auth_source` tagging
    - `COGNITO_SVC_USER_POOL_ID` / `COGNITO_SVC_JWKS_URL` / `COGNITO_SVC_APP_CLIENT_ID` configuration
    - Service-account lifecycle CLI (`cmd/auth_related/service.account.tool`)
    - Caller flow: `InitiateAuth` + bearer JWT, revocation strategy

## Quick Start

### For New Developers
1. Start with **[Getting Started Guide](05-getting-started.md)** for environment setup
2. Read **[Code Organization](06-code-organization.md)** to understand the codebase structure
3. Review **[Configuration Management](07-configuration-management.md)** for local development
4. Explore **[API Documentation](03-api-documentation.md)** for endpoint details

### For System Architects
1. Begin with **[System Overview](01-system-overview.md)** for architectural understanding
2. Study **[Service Architecture](02-service-architecture.md)** for microservices design
3. Review **[Data Architecture](04-data-architecture.md)** for database design
4. Consult **[Operations Guide](08-operations-guide.md)** for deployment planning

### For Operations Teams
1. Focus on **[Operations Guide](08-operations-guide.md)** for deployment and monitoring
2. Reference **[Configuration Management](07-configuration-management.md)** for environment setup
3. Use **[API Documentation](03-api-documentation.md)** for health checks and troubleshooting
4. Understand **[Service Architecture](02-service-architecture.md)** for system dependencies

## System Characteristics

### 🏗️ Architecture
- **Event-driven microservices** with SQS-based communication
- **Active services**: 1 HTTP API + 5 active runners
- **Compatibility services**: additional deprecated runner binaries may still be present in some deployments
- **Multi-tenant design** with client isolation
- **Comprehensive versioning** for configurations

### 🔧 Technology Stack
- **Backend**: Go 1.24.0 with Echo framework
- **Database**: PostgreSQL 17.2 with SQLC
- **Cloud**: AWS (S3, SQS, Cognito)
- **Containers**: Docker with multi-stage builds
- **Monitoring**: Prometheus + OpenTelemetry

### 📊 Key Capabilities
- **Document Processing**: Multi-format document upload (PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML), download, initialization, sync-gating, and cleaning
- **Multi-tenant Support**: Isolated client environments
- **Document Organization**: Folders and labels for document management
- **Export Operations**: Structured data export in multiple formats

## Documentation Tools

### PDF Generation

Generate professional PDF documentation with rendered Mermaid diagrams and working internal links:

```bash
# Navigate to the scripts directory
cd docs/ai.generated/scripts

# Run the PDF generation script
./build-and-generate-pdf.sh
```

**Output**: Timestamped PDF file (e.g., `DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf`) in the `docs/ai.generated/` directory.

**Prerequisites:**
- Docker installed and running
- Sufficient disk space (~50MB for build process)

**Features:**
- ✅ **Rendered Mermaid diagrams** - All diagram code blocks converted to PNG graphics
- ✅ **Working internal links** - Cross-references between sections function properly
- ✅ **Professional formatting** - Custom CSS styling with syntax highlighting
- ✅ **Page breaks** - Logical section separation for print/digital viewing
- ✅ **Table of contents** - Auto-generated with clickable navigation

**Technical Details:**
- Uses Docker container with Node.js 18, Chromium, and Mermaid CLI
- Requires `--security-opt seccomp=unconfined --cap-add=SYS_ADMIN` for Mermaid rendering
- Processes all documentation files in logical order with proper formatting
- Generates intermediate PNG files for diagrams (cleaned up automatically)

**Troubleshooting:**
- If Mermaid diagrams don't render, ensure Docker has sufficient privileges
- Large PDFs (>1MB) are normal due to embedded graphics and styling
- Permission errors: Run `chmod +x build-and-generate-pdf.sh`
- Container conflicts: `docker ps -a | grep doczy-docs-generator && docker rm -f doczy-docs-generator`

### Mermaid Diagram Link Generation

For programmatic generation of mermaid.live links in documentation, use the provided scripts:

[link to rendered version here](https://mermaid.live/edit#pako:eNqFkM1KBDEQBl-lyVydycHbCIKieBEU0VtAmqRnE8wfSUcc1n134zqCl8XrR3dR1F7oZEjMIAa4o0gFmQAhUAnozOTdO8HL0z0sJYW-G4e7ggEW50lFjfy7TNsHfMIkqy4uc5XbNnIaW_FTtSqqOMBDgVYJHle2KXZAIc1-PYXLx7Pz09i8_mBvPzBkT52Nuy5H2iZQotOyheebC7gax8trJf4z7H6Nc-MZLHOus5R_Y0gyjoeMb2mm-J3OvG7K4gzEdtlz7gVbCsewhhZsnsXh8AXqY4CE)

```bash
# Generate a mermaid.live URL from a diagram file
cat diagram.mermaid | ./scripts/mermaid-to-url.sh

# Or use Python directly
cat diagram.mermaid | python3 ./scripts/mermaid-to-url.py

# Example usage
echo "graph TD; A-->B" | ./scripts/mermaid-to-url.sh
# Output: https://mermaid.live/edit#pako:encoded_diagram
```

**Features:**
- **Pako compression**: Uses proper deflate compression for URL optimization
- **URL-safe encoding**: Compatible with mermaid.live format
- **Pipe-friendly**: Accepts mermaid syntax via stdin
- **Cross-platform**: Works with bash or Python 3

**Use cases:**
- Automated documentation generation
- CI/CD pipeline diagram embedding
- Dynamic diagram URL creation for wikis/docs

### Post-Processing Documentation with Mermaid Links

After generating documentation, you can automatically add rendered diagram links to all mermaid diagrams:

```bash
# Add mermaid.live links to all markdown files in a directory
python3 ./scripts/add-mermaid-links.py --target-dir ./docs

# Preview changes without modifying files
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --dry-run

# Process subdirectories recursively
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive
```

**Features:**
- **Automatic detection**: Finds mermaid diagrams by language tag or content keywords
- **Non-destructive**: Preserves all original content, only adds links
- **Duplicate prevention**: Skips diagrams that already have links
- **Flexible targeting**: Process single directories or recursive scans

**Workflow integration:**
1. Generate your documentation with embedded mermaid diagrams
2. Run the link addition script as a post-processing step
3. All mermaid diagrams now include clickable links to rendered versions

### Complete Documentation Workflow

For comprehensive documentation generation and maintenance:

```bash
# 1. Navigate to the ai.generated directory
cd docs/ai.generated

# 2. Generate updated PDF with rendered diagrams
cd scripts && ./build-and-generate-pdf.sh && cd ..

# 3. (Optional) Add mermaid.live links to markdown files
python3 ./scripts/add-mermaid-links.py --target-dir . --recursive

# 4. Verify output
ls -la DoczyAI-Documentation-*.pdf
```

This workflow ensures all documentation formats are current with proper diagram rendering and cross-references.

## Contributing

### Development Workflow
1. **Setup**: Follow the Getting Started Guide
2. **Development**: Use task-based workflows (`task --list`)
3. **Testing**: Maintain 80% total, 60% function coverage
4. **Documentation**: Update relevant docs with changes

### Code Standards
- **Language**: Go with standard project layout
- **Testing**: Table-driven tests with testcontainers
- **API**: Contract-first with OpenAPI generation
- **Database**: Migration-based schema evolution

## Support and Resources

### Internal Resources
- **Swagger UI**: http://localhost:8080/swagger/ (when running locally)
- **Metrics**: http://localhost:9091 (Prometheus)
- **Health Checks**: http://localhost:8080/health

### External Dependencies
- **AWS Services**: S3, SQS, Cognito
- **Database**: PostgreSQL 17.2+
- **Monitoring**: Prometheus-compatible metrics

---

*This documentation is automatically generated and maintained alongside the codebase. For questions or improvements, please contact the platform team.*