Merged in feature/ai-generated-docs (pull request #167)

ai generated docs and related tools for publishing new versions * ai generated docs and tooling * tools * add mermaid links * add task docs:ai:generate * Merged main into feature/ai-generated-docs Approved-by: Michael McGuinness
2025-06-24 19:56:56 +00:00
parent 079aec770d
commit 45b5f2fd50
16 changed files with 3717 additions and 0 deletions
@@ -0,0 +1,188 @@
+# DoczyAI Query Orchestration Platform Documentation
+
+## Overview
+
+This documentation provides comprehensive coverage of the DoczyAI Query Orchestration Platform - a cloud-native document processing and query orchestration system designed to extract structured information from documents using configurable, hierarchical queries.
+
+## Documentation Structure
+
+### 📋 System Documentation
+
+1. **[System Overview](01-system-overview.md)**
+   - Platform purpose and business capabilities
+   - Technology stack and architecture overview
+   - High-level component diagram
+   - Deployment architecture and scalability
+
+2. **[Service Architecture](02-service-architecture.md)**
+   - Detailed service catalog (12 services)
+   - Document processing pipeline flow
+   - Inter-service communication patterns
+   - Queue architecture and message formats
+
+3. **[API Documentation](03-api-documentation.md)**
+   - REST API endpoints and authentication
+   - Request/response schemas and validation
+   - OpenAPI specification and code generation
+   - Rate limiting and error handling
+
+4. **[Data Architecture](04-data-architecture.md)**
+   - Database schema and entity relationships
+   - Versioning system and temporal queries
+   - Query dependency management
+   - Performance optimization strategies
+
+### 🛠️ Developer Documentation
+
+5. **[Getting Started Guide](05-getting-started.md)**
+   - Prerequisites and local setup
+   - Development environment (Devbox)
+   - Common development workflows
+   - Testing and debugging guidelines
+
+6. **[Code Organization](06-code-organization.md)**
+   - Project structure and package architecture
+   - Design patterns and naming conventions
+   - Code style guidelines and best practices
+   - Testing patterns and examples
+
+7. **[Configuration Management](07-configuration-management.md)**
+   - Environment variable hierarchy
+   - Service configuration patterns
+   - Development vs production settings
+   - Security and validation practices
+
+8. **[Operations Guide](08-operations-guide.md)**
+   - Deployment procedures
+   - Monitoring and alerting setup
+   - Log analysis and debugging
+   - Backup, recovery, and capacity planning
+
+## Quick Start
+
+### For New Developers
+1. Start with **[Getting Started Guide](05-getting-started.md)** for environment setup
+2. Read **[Code Organization](06-code-organization.md)** to understand the codebase structure
+3. Review **[Configuration Management](07-configuration-management.md)** for local development
+4. Explore **[API Documentation](03-api-documentation.md)** for endpoint details
+
+### For System Architects
+1. Begin with **[System Overview](01-system-overview.md)** for architectural understanding
+2. Study **[Service Architecture](02-service-architecture.md)** for microservices design
+3. Review **[Data Architecture](04-data-architecture.md)** for database design
+4. Consult **[Operations Guide](08-operations-guide.md)** for deployment planning
+
+### For Operations Teams
+1. Focus on **[Operations Guide](08-operations-guide.md)** for deployment and monitoring
+2. Reference **[Configuration Management](07-configuration-management.md)** for environment setup
+3. Use **[API Documentation](03-api-documentation.md)** for health checks and troubleshooting
+4. Understand **[Service Architecture](02-service-architecture.md)** for system dependencies
+
+## System Characteristics
+
+### 🏗️ Architecture
+- **Event-driven microservices** with SQS-based communication
+- **12 distinct services**: 1 HTTP API + 8 runners + 3 utilities
+- **Multi-tenant design** with client isolation
+- **Comprehensive versioning** for queries and configurations
+
+### 🔧 Technology Stack
+- **Backend**: Go 1.24.0 with Echo framework
+- **Database**: PostgreSQL 17.2 with SQLC
+- **Cloud**: AWS (S3, SQS, Textract, Cognito)
+- **Containers**: Docker with multi-stage builds
+- **Monitoring**: Prometheus + OpenTelemetry
+
+### 📊 Key Capabilities
+- **Document Processing**: PDF upload, cleaning, and text extraction
+- **Query Orchestration**: Configurable data extraction with dependencies
+- **Multi-tenant Support**: Isolated client environments
+- **Export Operations**: Structured data export in multiple formats
+- **Version Management**: Full query and configuration versioning
+
+## Documentation Tools
+
+### Mermaid Diagram Link Generation
+
+For programmatic generation of mermaid.live links in documentation, use the provided scripts:
+
+[link to rendered version here](https://mermaid.live/edit#pako:eNqFkM1KBDEQBl-lyVydycHbCIKieBEU0VtAmqRnE8wfSUcc1n134zqCl8XrR3dR1F7oZEjMIAa4o0gFmQAhUAnozOTdO8HL0z0sJYW-G4e7ggEW50lFjfy7TNsHfMIkqy4uc5XbNnIaW_FTtSqqOMBDgVYJHle2KXZAIc1-PYXLx7Pz09i8_mBvPzBkT52Nuy5H2iZQotOyheebC7gax8trJf4z7H6Nc-MZLHOus5R_Y0gyjoeMb2mm-J3OvG7K4gzEdtlz7gVbCsewhhZsnsXh8AXqY4CE)
+
+```bash
+# Generate a mermaid.live URL from a diagram file
+cat diagram.mermaid | ./scripts/mermaid-to-url.sh
+
+# Or use Python directly
+cat diagram.mermaid | python3 ./scripts/mermaid-to-url.py
+
+# Example usage
+echo "graph TD; A-->B" | ./scripts/mermaid-to-url.sh
+# Output: https://mermaid.live/edit#pako:encoded_diagram
+```
+
+**Features:**
+- **Pako compression**: Uses proper deflate compression for URL optimization
+- **URL-safe encoding**: Compatible with mermaid.live format
+- **Pipe-friendly**: Accepts mermaid syntax via stdin
+- **Cross-platform**: Works with bash or Python 3
+
+**Use cases:**
+- Automated documentation generation
+- CI/CD pipeline diagram embedding
+- Dynamic diagram URL creation for wikis/docs
+
+### Post-Processing Documentation with Mermaid Links
+
+After generating documentation, you can automatically add rendered diagram links to all mermaid diagrams:
+
+```bash
+# Add mermaid.live links to all markdown files in a directory
+python3 ./scripts/add-mermaid-links.py --target-dir ./docs
+
+# Preview changes without modifying files
+python3 ./scripts/add-mermaid-links.py --target-dir ./docs --dry-run
+
+# Process subdirectories recursively
+python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive
+```
+
+**Features:**
+- **Automatic detection**: Finds mermaid diagrams by language tag or content keywords
+- **Non-destructive**: Preserves all original content, only adds links
+- **Duplicate prevention**: Skips diagrams that already have links
+- **Flexible targeting**: Process single directories or recursive scans
+
+**Workflow integration:**
+1. Generate your documentation with embedded mermaid diagrams
+2. Run the link addition script as a post-processing step
+3. All mermaid diagrams now include clickable links to rendered versions
+
+## Contributing
+
+### Development Workflow
+1. **Setup**: Follow the Getting Started Guide
+2. **Development**: Use task-based workflows (`task --list`)
+3. **Testing**: Maintain 80% total, 60% function coverage
+4. **Documentation**: Update relevant docs with changes
+
+### Code Standards
+- **Language**: Go with standard project layout
+- **Testing**: Table-driven tests with testcontainers
+- **API**: Contract-first with OpenAPI generation
+- **Database**: Migration-based schema evolution
+
+## Support and Resources
+
+### Internal Resources
+- **Swagger UI**: http://localhost:8080/swagger/ (when running locally)
+- **Metrics**: http://localhost:9091 (Prometheus)
+- **Health Checks**: http://localhost:8080/health
+
+### External Dependencies
+- **AWS Services**: S3, SQS, Textract, Cognito
+- **Database**: PostgreSQL 17.2+
+- **Monitoring**: Prometheus-compatible metrics
+
+---
+
+*This documentation is automatically generated and maintained alongside the codebase. For questions or improvements, please contact the platform team.*