Jay Brown
451da3d26d
Support for service accounts with static credentials * feature complete * tests and docs * lint fix
296 lines
12 KiB
Markdown
296 lines
12 KiB
Markdown
# DoczyAI Document Processing Platform Documentation
|
|
|
|
## Overview
|
|
|
|
This documentation provides comprehensive coverage of the DoczyAI Document Processing Platform - a cloud-native document processing system designed to extract structured information from documents.
|
|
|
|
## Architecture Diagram
|
|
|
|
The diagram below shows the full API layer architecture — from HTTP endpoints through middleware, controllers, services, and the repository layer, ending at the database and external services. Request flow runs left to right.
|
|
|
|
[](architecture-diagram.svg)
|
|
|
|
*[View scalable SVG version](architecture-diagram.svg)*
|
|
|
|
## Documentation Structure
|
|
|
|
### 📋 System Documentation
|
|
|
|
1. **[System Overview](01-system-overview.md)**
|
|
- Platform purpose and business capabilities
|
|
- Technology stack and architecture overview
|
|
- High-level component diagram
|
|
- Deployment architecture and scalability
|
|
|
|
2. **[Service Architecture](02-service-architecture.md)**
|
|
- Detailed service catalog (active + compatibility services)
|
|
- Document processing pipeline flow
|
|
- Inter-service communication patterns
|
|
- Queue architecture and message formats
|
|
|
|
3. **[API Documentation](03-api-documentation.md)**
|
|
- REST API endpoints and authentication
|
|
- Request/response schemas and validation
|
|
- OpenAPI specification and code generation
|
|
- Rate limiting and error handling
|
|
|
|
4. **[Data Architecture](04-data-architecture.md)**
|
|
- Database schema and entity relationships
|
|
- Versioning system
|
|
- Document organization (folders and labels)
|
|
- Performance optimization strategies
|
|
|
|
### 🛠️ Developer Documentation
|
|
|
|
5. **[Getting Started Guide](05-getting-started.md)**
|
|
- Prerequisites and local setup
|
|
- Development environment (Devbox)
|
|
- Common development workflows
|
|
- Testing and debugging guidelines
|
|
|
|
6. **[Code Organization](06-code-organization.md)**
|
|
- Project structure and package architecture
|
|
- Design patterns and naming conventions
|
|
- Code style guidelines and best practices
|
|
- Testing patterns and examples
|
|
|
|
7. **[Configuration Management](07-configuration-management.md)**
|
|
- Environment variable hierarchy
|
|
- Service configuration patterns
|
|
- Development vs production settings
|
|
- Security and validation practices
|
|
|
|
8. **[Operations Guide](08-operations-guide.md)**
|
|
- Deployment procedures
|
|
- Monitoring and alerting setup
|
|
- Log analysis and debugging
|
|
- Backup, recovery, and capacity planning
|
|
|
|
9. **[Authentication Guide](09-authentication-guide.md)**
|
|
- JWT authentication with AWS Cognito
|
|
- Token types and refresh strategy
|
|
- CORS configuration
|
|
|
|
10. **[Appendix](10-appendix.md)**
|
|
- S3 Storage Guide: BucketKey path format, segment reference, partitioning
|
|
- Watching S3 for client document imports
|
|
- Document recovery from S3 buckets
|
|
|
|
11. **[Custom Metadata Guide](11-custom-metadata-guide.md)**
|
|
- Overview: custom schemas vs legacy field extractions
|
|
- Authorization matrix (super_admin, user_admin, client_user, auditor)
|
|
- Platform operator workflow: define schema, assign to document
|
|
- End-user workflow: write, read current, read by version, read history
|
|
- Schema definition requirements and sample schemas
|
|
- Schema versioning: how POST .../versions works, status lifecycle
|
|
- Reset-metadata runbook: schema upgrade flow with metadata wipe warning
|
|
- Cascade deletion behavior (no special steps needed)
|
|
- Permit.io deployment note
|
|
|
|
**[File Validity Checks](file.filters.md)**
|
|
- PDF-specific validation (signature, structure, per-page size, dimensions, DPI) and SHA-256 deduplication
|
|
- Per-format checks for DOCX, XLSX, PPTX, TIFF, JPEG, PNG, BMP, TXT, EML
|
|
- Common pre-pipeline checks (extension allowlist, ZIP-archive limits, OLE2-encryption sniff)
|
|
- Failure-reason catalog (`InvalidDocumentReason`)
|
|
|
|
12. **[Chatbot Guide](12-chatbot-guide.md)**
|
|
- Overview: queryAPI's facade over the Aaria FastAPI agent service
|
|
- Architecture: `bot_*` data model, AddTurn tx1+FastAPI+tx2 flow, state stripping
|
|
- Configuration reference: 5 `CHATBOT_*` env vars, derived values, validation
|
|
- API reference: 8 user routes + 3 super-admin routes; HTTP status mapping
|
|
- Operator runbook: failed-turn classifications (`errored`, `abandoned`, `session_deleted`) and recovery paths
|
|
- Rate-limit overrides: M6 behavior change and Echo `:param` template form
|
|
- Known warts and tradeoffs (CHECK guard, error chaining, label-record divergence, etc.)
|
|
|
|
13. **[Two-Pool Service Accounts](13-two-pool-service-accounts.md)**
|
|
- Multi-issuer JWT validation (pool A + pool B) and `auth_source` tagging
|
|
- `COGNITO_SVC_USER_POOL_ID` / `COGNITO_SVC_JWKS_URL` / `COGNITO_SVC_APP_CLIENT_ID` configuration
|
|
- Service-account lifecycle CLI (`cmd/auth_related/service.account.tool`)
|
|
- Caller flow: `InitiateAuth` + bearer JWT, revocation strategy
|
|
|
|
## Quick Start
|
|
|
|
### For New Developers
|
|
1. Start with **[Getting Started Guide](05-getting-started.md)** for environment setup
|
|
2. Read **[Code Organization](06-code-organization.md)** to understand the codebase structure
|
|
3. Review **[Configuration Management](07-configuration-management.md)** for local development
|
|
4. Explore **[API Documentation](03-api-documentation.md)** for endpoint details
|
|
|
|
### For System Architects
|
|
1. Begin with **[System Overview](01-system-overview.md)** for architectural understanding
|
|
2. Study **[Service Architecture](02-service-architecture.md)** for microservices design
|
|
3. Review **[Data Architecture](04-data-architecture.md)** for database design
|
|
4. Consult **[Operations Guide](08-operations-guide.md)** for deployment planning
|
|
|
|
### For Operations Teams
|
|
1. Focus on **[Operations Guide](08-operations-guide.md)** for deployment and monitoring
|
|
2. Reference **[Configuration Management](07-configuration-management.md)** for environment setup
|
|
3. Use **[API Documentation](03-api-documentation.md)** for health checks and troubleshooting
|
|
4. Understand **[Service Architecture](02-service-architecture.md)** for system dependencies
|
|
|
|
## System Characteristics
|
|
|
|
### 🏗️ Architecture
|
|
- **Event-driven microservices** with SQS-based communication
|
|
- **Active services**: 1 HTTP API + 5 active runners
|
|
- **Compatibility services**: additional deprecated runner binaries may still be present in some deployments
|
|
- **Multi-tenant design** with client isolation
|
|
- **Comprehensive versioning** for configurations
|
|
|
|
### 🔧 Technology Stack
|
|
- **Backend**: Go 1.24.0 with Echo framework
|
|
- **Database**: PostgreSQL 17.2 with SQLC
|
|
- **Cloud**: AWS (S3, SQS, Cognito)
|
|
- **Containers**: Docker with multi-stage builds
|
|
- **Monitoring**: Prometheus + OpenTelemetry
|
|
|
|
### 📊 Key Capabilities
|
|
- **Document Processing**: Multi-format document upload (PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML), download, initialization, sync-gating, and cleaning
|
|
- **Multi-tenant Support**: Isolated client environments
|
|
- **Document Organization**: Folders and labels for document management
|
|
- **Export Operations**: Structured data export in multiple formats
|
|
|
|
## Documentation Tools
|
|
|
|
### PDF Generation
|
|
|
|
Generate professional PDF documentation with rendered Mermaid diagrams and working internal links:
|
|
|
|
```bash
|
|
# Navigate to the scripts directory
|
|
cd docs/ai.generated/scripts
|
|
|
|
# Run the PDF generation script
|
|
./build-and-generate-pdf.sh
|
|
```
|
|
|
|
**Output**: Timestamped PDF file (e.g., `DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf`) in the `docs/ai.generated/` directory.
|
|
|
|
**Prerequisites:**
|
|
- Docker installed and running
|
|
- Sufficient disk space (~50MB for build process)
|
|
|
|
**Features:**
|
|
- ✅ **Rendered Mermaid diagrams** - All diagram code blocks converted to PNG graphics
|
|
- ✅ **Working internal links** - Cross-references between sections function properly
|
|
- ✅ **Professional formatting** - Custom CSS styling with syntax highlighting
|
|
- ✅ **Page breaks** - Logical section separation for print/digital viewing
|
|
- ✅ **Table of contents** - Auto-generated with clickable navigation
|
|
|
|
**Technical Details:**
|
|
- Uses Docker container with Node.js 18, Chromium, and Mermaid CLI
|
|
- Requires `--security-opt seccomp=unconfined --cap-add=SYS_ADMIN` for Mermaid rendering
|
|
- Processes all documentation files in logical order with proper formatting
|
|
- Generates intermediate PNG files for diagrams (cleaned up automatically)
|
|
|
|
**Troubleshooting:**
|
|
- If Mermaid diagrams don't render, ensure Docker has sufficient privileges
|
|
- Large PDFs (>1MB) are normal due to embedded graphics and styling
|
|
- Permission errors: Run `chmod +x build-and-generate-pdf.sh`
|
|
- Container conflicts: `docker ps -a | grep doczy-docs-generator && docker rm -f doczy-docs-generator`
|
|
|
|
### Mermaid Diagram Link Generation
|
|
|
|
For programmatic generation of mermaid.live links in documentation, use the provided scripts:
|
|
|
|
[link to rendered version here](https://mermaid.live/edit#pako:eNqFkM1KBDEQBl-lyVydycHbCIKieBEU0VtAmqRnE8wfSUcc1n134zqCl8XrR3dR1F7oZEjMIAa4o0gFmQAhUAnozOTdO8HL0z0sJYW-G4e7ggEW50lFjfy7TNsHfMIkqy4uc5XbNnIaW_FTtSqqOMBDgVYJHle2KXZAIc1-PYXLx7Pz09i8_mBvPzBkT52Nuy5H2iZQotOyheebC7gax8trJf4z7H6Nc-MZLHOus5R_Y0gyjoeMb2mm-J3OvG7K4gzEdtlz7gVbCsewhhZsnsXh8AXqY4CE)
|
|
|
|
```bash
|
|
# Generate a mermaid.live URL from a diagram file
|
|
cat diagram.mermaid | ./scripts/mermaid-to-url.sh
|
|
|
|
# Or use Python directly
|
|
cat diagram.mermaid | python3 ./scripts/mermaid-to-url.py
|
|
|
|
# Example usage
|
|
echo "graph TD; A-->B" | ./scripts/mermaid-to-url.sh
|
|
# Output: https://mermaid.live/edit#pako:encoded_diagram
|
|
```
|
|
|
|
**Features:**
|
|
- **Pako compression**: Uses proper deflate compression for URL optimization
|
|
- **URL-safe encoding**: Compatible with mermaid.live format
|
|
- **Pipe-friendly**: Accepts mermaid syntax via stdin
|
|
- **Cross-platform**: Works with bash or Python 3
|
|
|
|
**Use cases:**
|
|
- Automated documentation generation
|
|
- CI/CD pipeline diagram embedding
|
|
- Dynamic diagram URL creation for wikis/docs
|
|
|
|
### Post-Processing Documentation with Mermaid Links
|
|
|
|
After generating documentation, you can automatically add rendered diagram links to all mermaid diagrams:
|
|
|
|
```bash
|
|
# Add mermaid.live links to all markdown files in a directory
|
|
python3 ./scripts/add-mermaid-links.py --target-dir ./docs
|
|
|
|
# Preview changes without modifying files
|
|
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --dry-run
|
|
|
|
# Process subdirectories recursively
|
|
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive
|
|
```
|
|
|
|
**Features:**
|
|
- **Automatic detection**: Finds mermaid diagrams by language tag or content keywords
|
|
- **Non-destructive**: Preserves all original content, only adds links
|
|
- **Duplicate prevention**: Skips diagrams that already have links
|
|
- **Flexible targeting**: Process single directories or recursive scans
|
|
|
|
**Workflow integration:**
|
|
1. Generate your documentation with embedded mermaid diagrams
|
|
2. Run the link addition script as a post-processing step
|
|
3. All mermaid diagrams now include clickable links to rendered versions
|
|
|
|
### Complete Documentation Workflow
|
|
|
|
For comprehensive documentation generation and maintenance:
|
|
|
|
```bash
|
|
# 1. Navigate to the ai.generated directory
|
|
cd docs/ai.generated
|
|
|
|
# 2. Generate updated PDF with rendered diagrams
|
|
cd scripts && ./build-and-generate-pdf.sh && cd ..
|
|
|
|
# 3. (Optional) Add mermaid.live links to markdown files
|
|
python3 ./scripts/add-mermaid-links.py --target-dir . --recursive
|
|
|
|
# 4. Verify output
|
|
ls -la DoczyAI-Documentation-*.pdf
|
|
```
|
|
|
|
This workflow ensures all documentation formats are current with proper diagram rendering and cross-references.
|
|
|
|
## Contributing
|
|
|
|
### Development Workflow
|
|
1. **Setup**: Follow the Getting Started Guide
|
|
2. **Development**: Use task-based workflows (`task --list`)
|
|
3. **Testing**: Maintain 80% total, 60% function coverage
|
|
4. **Documentation**: Update relevant docs with changes
|
|
|
|
### Code Standards
|
|
- **Language**: Go with standard project layout
|
|
- **Testing**: Table-driven tests with testcontainers
|
|
- **API**: Contract-first with OpenAPI generation
|
|
- **Database**: Migration-based schema evolution
|
|
|
|
## Support and Resources
|
|
|
|
### Internal Resources
|
|
- **Swagger UI**: http://localhost:8080/swagger/ (when running locally)
|
|
- **Metrics**: http://localhost:9091 (Prometheus)
|
|
- **Health Checks**: http://localhost:8080/health
|
|
|
|
### External Dependencies
|
|
- **AWS Services**: S3, SQS, Cognito
|
|
- **Database**: PostgreSQL 17.2+
|
|
- **Monitoring**: Prometheus-compatible metrics
|
|
|
|
---
|
|
|
|
*This documentation is automatically generated and maintained alongside the codebase. For questions or improvements, please contact the platform team.*
|