diff --git a/docs/ai.generated/01-system-overview.md b/docs/ai.generated/01-system-overview.md index 45e87296..225d8e81 100644 --- a/docs/ai.generated/01-system-overview.md +++ b/docs/ai.generated/01-system-overview.md @@ -7,20 +7,23 @@ DoczyAI is a cloud-native document processing and query orchestration platform d ### Core Business Capabilities - **Multi-tenant Document Processing**: Isolated processing environments per client +- **Batch Document Processing**: ZIP archive uploads for processing multiple documents simultaneously - **Configurable Query System**: Custom data extraction rules with dependency management - **Automated Processing Pipeline**: Seamless flow from document upload to structured output - **Version Management**: Complete versioning for queries and configurations - **Export and Integration**: Structured data export capabilities for downstream systems +- **Enhanced Observability**: Comprehensive metrics and monitoring with Prometheus integration ### Technology Stack -- **Language**: Go 1.24.0 +- **Language**: Go 1.24.3 - **Web Framework**: Echo v4 with OpenAPI code generation - **Database**: PostgreSQL 17.2 with SQLC for type-safe queries - **Messaging**: AWS SQS for event-driven microservices - **Storage**: AWS S3 for document storage - **Text Processing**: AWS Textract for OCR and text extraction - **Authentication**: AWS Cognito with JWT-based authorization -- **Monitoring**: Prometheus metrics and OpenTelemetry tracing +- **Monitoring**: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing +- **Authorization**: Permit.io RBAC integration for fine-grained permissions - **Container Platform**: Docker with multi-stage builds ## Architecture Overview @@ -28,8 +31,9 @@ DoczyAI is a cloud-native document processing and query orchestration platform d ### System Architecture Pattern The system follows an **event-driven microservices architecture** with the following characteristics: - **12 distinct services**: 1 HTTP API + 8 queue-based runners + 3 utility services -- **Asynchronous communication**: SQS queues between services -- **State management**: PostgreSQL as central data store +- **Asynchronous communication**: SQS queues between services with batch processing capabilities +- **Background Task Framework**: Comprehensive background task execution with periodic scheduling +- **State management**: PostgreSQL as central data store with enhanced schema for batch operations - **External service integration**: AWS managed services for cloud capabilities ### High-Level Component Diagram @@ -108,11 +112,11 @@ graph TB - **Processing Scaling**: Auto-scaling based on SQS queue metrics ### Security Architecture -- **Authentication**: OAuth2 with AWS Cognito integration -- **Authorization**: Role-based access control with group-based permissions -- **Data Protection**: Encrypted storage (S3 server-side encryption) +- **Authentication**: OAuth2 with AWS Cognito integration and MFA support +- **Authorization**: Enhanced RBAC with Permit.io integration and group-based permissions +- **Data Protection**: Encrypted storage (S3 server-side encryption) with folder path preservation - **Network Security**: VPC isolation and security groups -- **API Security**: JWT validation and OpenAPI request validation +- **API Security**: JWT validation and OpenAPI request validation with enhanced token handling ## Integration Points diff --git a/docs/ai.generated/02-service-architecture.md b/docs/ai.generated/02-service-architecture.md index 9110df80..fab915e9 100644 --- a/docs/ai.generated/02-service-architecture.md +++ b/docs/ai.generated/02-service-architecture.md @@ -10,18 +10,20 @@ - **Type**: HTTP REST API - **Key Responsibilities**: - Client management and authentication - - Document upload and metadata management + - Document upload and metadata management (single and batch) + - Batch upload processing with ZIP archive support - Query definition and testing - Export operations and status monitoring + - Background task coordination and monitoring - User interface and API documentation #### storeEventRunner - S3 Event Processor - **Purpose**: Entry point for document processing pipeline - **Port**: 8081 - **Type**: Event-driven queue consumer -- **Input**: S3 PUT event notifications +- **Input**: S3 PUT event notifications (single files and batch uploads) - **Output**: Events to DOCINIT queue -- **Responsibilities**: Processes S3 upload events and initiates document processing +- **Responsibilities**: Processes S3 upload events, handles ZIP archive extraction, initiates document processing #### docInitRunner - Document Initialization - **Purpose**: Document registration and duplicate detection @@ -103,8 +105,19 @@ - **Use Case**: Development and testing workflows #### metricsExample_test - Monitoring Example -- **Purpose**: Demonstrates Prometheus metrics implementation +- **Purpose**: Demonstrates comprehensive Prometheus metrics implementation - **Type**: Educational/testing service +- **Features**: Showcases 12 standard metric types (Counter, Gauge, Histogram, Summary) + +#### Background Task Framework +- **Purpose**: Comprehensive background task execution system +- **Type**: Embedded framework within services +- **Features**: + - Periodic task execution with configurable intervals + - On-demand task triggering via signals + - Work coalescing to prevent duplicate processing + - Statistics tracking and performance monitoring + - Graceful shutdown with task completion ## Data Flow Architecture @@ -182,6 +195,7 @@ All services communicate through **AWS SQS queues** with the following patterns: | QUERY | querySyncRunner, queryRunner | queryRunner | `{document_id, query_id}` | | CLIENTSYNC | queryVersionSyncRunner | clientSyncRunner | `{client_id}` | | QUERYVERSIONSYNC | queryAPI | queryVersionSyncRunner | `{query_id}` | +| BATCH_PROCESSING | queryAPI (batch worker) | Internal batch processor | `{batch_id, files[]}` | ### Database Access Patterns - **Repository Pattern**: SQLC-generated type-safe database access @@ -204,7 +218,10 @@ All services communicate through **AWS SQS queues** with the following patterns: - **Compression**: Gzip compression for API responses ### Monitoring and Observability -- **Metrics**: Prometheus metrics from all services -- **Tracing**: OpenTelemetry distributed tracing -- **Logging**: Structured logging with correlation IDs -- **Health Checks**: Built-in health endpoints \ No newline at end of file +- **Metrics**: Comprehensive Prometheus metrics with 12 standard types (Counter, Gauge, Histogram, Summary) + - Request metrics, performance tracking, resource usage, dependency health + - Cache operations, background tasks, queue metrics, error tracking +- **Tracing**: OpenTelemetry distributed tracing with correlation IDs +- **Logging**: Structured logging with comprehensive correlation +- **Health Checks**: Built-in health endpoints with dependency validation +- **Background Task Monitoring**: Task execution statistics and performance tracking \ No newline at end of file diff --git a/docs/ai.generated/03-api-documentation.md b/docs/ai.generated/03-api-documentation.md index e449855a..a600670e 100644 --- a/docs/ai.generated/03-api-documentation.md +++ b/docs/ai.generated/03-api-documentation.md @@ -175,6 +175,83 @@ Retrieves document metadata by ID. **Path Parameters**: - `id`: Document UUID +### Batch Document Operations + +#### `POST /client/{id}/document/batch` +Uploads a ZIP archive containing multiple PDF documents for asynchronous batch processing. + +**Security**: Requires `uploaders` or `querybuilders` permission +**Content Type**: `multipart/form-data` +**Max Archive Size**: 1GB +**Supported Archive Type**: ZIP only + +**Form Parameters**: +- `archive`: ZIP file containing PDF documents (required) + +**Response** (202 Accepted): +```json +{ + "batch_id": "uuid", + "status": "processing", + "status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a" +} +``` + +#### `GET /client/{id}/document/batch` +Lists batch uploads for a client with pagination support. + +**Query Parameters**: +- `limit`: Maximum results (integer, default: 20) +- `offset`: Pagination offset (integer, default: 0) + +**Response**: +```json +{ + "batches": [ + { + "batch_id": "uuid", + "status": "completed" | "processing" | "failed" | "cancelled", + "total_documents": 15, + "processed_documents": 15, + "failed_documents": 0, + "created_at": "2024-01-01T00:00:00Z", + "completed_at": "2024-01-01T00:05:00Z" + } + ], + "total": 42 +} +``` + +#### `GET /client/{id}/document/batch/{batch_id}` +Retrieves detailed status information for a specific batch upload. + +**Path Parameters**: +- `batch_id`: Batch upload identifier (UUID) + +**Response**: +```json +{ + "batch_id": "uuid", + "status": "completed", + "total_documents": 15, + "processed_documents": 15, + "failed_documents": 0, + "failed_filenames": [], + "created_at": "2024-01-01T00:00:00Z", + "completed_at": "2024-01-01T00:05:00Z" +} +``` + +#### `DELETE /client/{id}/document/batch/{batch_id}` +Cancels a batch upload that is currently processing. + +**Path Parameters**: +- `batch_id`: Batch upload identifier (UUID) + +**Response**: +- `204`: Batch upload cancelled successfully +- `409`: Cannot cancel completed or already cancelled batch + ### Query Management #### `GET /query` diff --git a/docs/ai.generated/04-data-architecture.md b/docs/ai.generated/04-data-architecture.md index 1f99d1e6..7e21342e 100644 --- a/docs/ai.generated/04-data-architecture.md +++ b/docs/ai.generated/04-data-architecture.md @@ -29,6 +29,7 @@ CREATE TABLE documents ( id uuid PRIMARY KEY DEFAULT uuid_generate_v7(), clientId varchar(256) NOT NULL REFERENCES clients(clientId), hash varchar(256) NOT NULL, + filename varchar(512), UNIQUE(clientId, hash) ); ``` @@ -37,8 +38,38 @@ CREATE TABLE documents ( - UUID v7 primary keys for time-ordered insertion - Client isolation with foreign key constraints - Hash-based deduplication within client scope +- Filename preservation for folder path support - Supports document processing pipeline +### Batch Uploads (`batch_uploads`) +Tracks batch document upload operations with ZIP archive processing. + +```sql +CREATE TYPE batchstatus AS ENUM ('processing', 'completed', 'failed', 'cancelled'); + +CREATE TABLE batch_uploads ( + id uuid PRIMARY KEY DEFAULT uuid_generate_v7(), + clientId varchar(256) NOT NULL REFERENCES clients(clientId), + status batchstatus NOT NULL DEFAULT 'processing', + total_documents integer NOT NULL DEFAULT 0, + processed_documents integer NOT NULL DEFAULT 0, + failed_documents integer NOT NULL DEFAULT 0, + failed_filenames text[], + s3_bucket varchar(256), + s3_key varchar(512), + created_at timestamp NOT NULL DEFAULT NOW(), + completed_at timestamp +); +``` + +**Batch Processing Features**: +- UUID v7 for time-ordered batch tracking +- Comprehensive status tracking with ENUM types +- Progress counters for processed/failed documents +- Failed filename tracking for error reporting +- S3 storage metadata for archive management +- Temporal tracking with creation and completion timestamps + ### Queries (`queries`) Defines data extraction logic with type-specific implementations. diff --git a/docs/ai.generated/05-getting-started.md b/docs/ai.generated/05-getting-started.md index faff9856..013b7d9a 100644 --- a/docs/ai.generated/05-getting-started.md +++ b/docs/ai.generated/05-getting-started.md @@ -3,12 +3,13 @@ ## Prerequisites ### Required Tools -- **Devbox**: Provides reproducible development environment +- **Devbox**: Provides reproducible development environment with Go 1.24.3 - Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash` - NixOS: `devbox` (already available) - **Docker**: Container runtime for services and testing - Install from: https://docs.docker.com/engine/install/ - **Git**: Version control (usually pre-installed) +- **AWS CLI**: Required for Textract credentials and S3 access (installed via devbox) ### System Requirements - **OS**: Linux, macOS, or Windows with WSL2 diff --git a/docs/ai.generated/06-code-organization.md b/docs/ai.generated/06-code-organization.md index d57adafd..fd2f5d06 100644 --- a/docs/ai.generated/06-code-organization.md +++ b/docs/ai.generated/06-code-organization.md @@ -2,6 +2,8 @@ ## Project Structure +The project follows Go Standard Project Layout with enhanced organization for batch processing, observability, and background task management. + The codebase follows the [Standard Go Project Layout](https://github.com/golang-standards/project-layout) with additional conventions specific to the domain requirements. ### Directory Overview @@ -39,6 +41,15 @@ cmd/ ### Internal Packages (`internal/`) +Core business logic and infrastructure components, organized by domain: + +#### New Components (Since June 2025): +- **`internal/backgroundtask/`**: Background task execution framework with periodic scheduling +- **`internal/document/batch/`**: Batch upload processing and ZIP archive handling +- **`internal/serviceconfig/observability/prometheus/`**: Comprehensive Prometheus metrics framework + +### Internal Packages (`internal/`) + Core business logic organized by domain boundaries: #### Domain Services diff --git a/docs/ai.generated/07-configuration-management.md b/docs/ai.generated/07-configuration-management.md index 6396c4c9..d7452484 100644 --- a/docs/ai.generated/07-configuration-management.md +++ b/docs/ai.generated/07-configuration-management.md @@ -1,5 +1,9 @@ # Configuration Management +## Updated Configuration (September 2025) + +The system has expanded configuration support for batch processing, enhanced observability, and Permit.io RBAC integration. + ## Configuration Architecture The system implements a hierarchical, environment-driven configuration system designed for flexibility, type safety, and testability. Configuration is managed through the `internal/serviceconfig` package with domain-specific sub-packages. @@ -142,7 +146,7 @@ func (c Config) LoadAWSConfig(ctx context.Context) (aws.Config, error) { ### Queue Configuration (`serviceconfig/queue/`) ```go -// Base queue configuration +// Base queue configuration with batch processing support type Config struct { // Connection QueueURL string `env:"QUEUE_URL"` @@ -150,10 +154,11 @@ type Config struct { WaitTimeSeconds int32 `env:"QUEUE_WAIT_TIME" envDefault:"20"` VisibilityTimeout int32 `env:"QUEUE_VISIBILITY_TIMEOUT" envDefault:"30"` - // Processing + // Processing (enhanced for batch operations) WorkerCount int `env:"QUEUE_WORKER_COUNT" envDefault:"5"` MaxRetries int `env:"QUEUE_MAX_RETRIES" envDefault:"3"` RetryDelay int `env:"QUEUE_RETRY_DELAY" envDefault:"5"` + BatchSize int `env:"QUEUE_BATCH_SIZE" envDefault:"10"` } // Service-specific queue configs diff --git a/docs/ai.generated/08-operations-guide.md b/docs/ai.generated/08-operations-guide.md index b52bab6f..c2086141 100644 --- a/docs/ai.generated/08-operations-guide.md +++ b/docs/ai.generated/08-operations-guide.md @@ -1,5 +1,9 @@ # Operations Guide +## Updated Operations (September 2025) + +This guide covers deployment, monitoring, and operational procedures for the enhanced platform including batch processing capabilities, comprehensive Prometheus metrics, and background task management. + ## Deployment Procedures ### Local Development Deployment @@ -84,6 +88,10 @@ kubectl rollout undo deployment/query-api ## Monitoring and Alerting +### Enhanced Prometheus Metrics (September 2025) + +The platform now provides comprehensive observability with 12 standard metric types covering all aspects of system performance and health. + ### Prometheus Metrics #### System Metrics @@ -92,6 +100,10 @@ All services automatically expose metrics at `/metrics`: - **Database connections**: Pool usage, connection count, query duration - **Queue processing**: Message count, processing time, errors - **Business metrics**: Documents processed, queries executed, results generated +- **Batch processing**: ZIP upload metrics, batch completion rates, failed document tracking +- **Background tasks**: Task execution statistics, periodic job performance +- **Resource usage**: Memory, CPU, goroutine counts +- **Cache operations**: Hit/miss ratios, cache size, eviction rates #### Custom Metrics Example ```go diff --git a/docs/ai.generated/README.md b/docs/ai.generated/README.md index cc32ec30..aed6c649 100644 --- a/docs/ai.generated/README.md +++ b/docs/ai.generated/README.md @@ -102,6 +102,43 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr ## Documentation Tools +### PDF Generation + +Generate professional PDF documentation with rendered Mermaid diagrams and working internal links: + +```bash +# Navigate to the scripts directory +cd docs/ai.generated/scripts + +# Run the PDF generation script +./build-and-generate-pdf.sh +``` + +**Output**: Timestamped PDF file (e.g., `DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf`) in the `docs/ai.generated/` directory. + +**Prerequisites:** +- Docker installed and running +- Sufficient disk space (~50MB for build process) + +**Features:** +- ✅ **Rendered Mermaid diagrams** - All diagram code blocks converted to PNG graphics +- ✅ **Working internal links** - Cross-references between sections function properly +- ✅ **Professional formatting** - Custom CSS styling with syntax highlighting +- ✅ **Page breaks** - Logical section separation for print/digital viewing +- ✅ **Table of contents** - Auto-generated with clickable navigation + +**Technical Details:** +- Uses Docker container with Node.js 18, Chromium, and Mermaid CLI +- Requires `--security-opt seccomp=unconfined --cap-add=SYS_ADMIN` for Mermaid rendering +- Processes all documentation files in logical order with proper formatting +- Generates intermediate PNG files for diagrams (cleaned up automatically) + +**Troubleshooting:** +- If Mermaid diagrams don't render, ensure Docker has sufficient privileges +- Large PDFs (>1MB) are normal due to embedded graphics and styling +- Permission errors: Run `chmod +x build-and-generate-pdf.sh` +- Container conflicts: `docker ps -a | grep doczy-docs-generator && docker rm -f doczy-docs-generator` + ### Mermaid Diagram Link Generation For programmatic generation of mermaid.live links in documentation, use the provided scripts: @@ -157,6 +194,26 @@ python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive 2. Run the link addition script as a post-processing step 3. All mermaid diagrams now include clickable links to rendered versions +### Complete Documentation Workflow + +For comprehensive documentation generation and maintenance: + +```bash +# 1. Navigate to the ai.generated directory +cd docs/ai.generated + +# 2. Generate updated PDF with rendered diagrams +cd scripts && ./build-and-generate-pdf.sh && cd .. + +# 3. (Optional) Add mermaid.live links to markdown files +python3 ./scripts/add-mermaid-links.py --target-dir . --recursive + +# 4. Verify output +ls -la DoczyAI-Documentation-*.pdf +``` + +This workflow ensures all documentation formats are current with proper diagram rendering and cross-references. + ## Contributing ### Development Workflow diff --git a/docs/ai.generated/scripts/Dockerfile b/docs/ai.generated/scripts/Dockerfile new file mode 100644 index 00000000..b349fcc5 --- /dev/null +++ b/docs/ai.generated/scripts/Dockerfile @@ -0,0 +1,40 @@ +# Dockerfile for markdown to PDF conversion with Mermaid support +FROM node:18-alpine + +# Install chromium and required dependencies for puppeteer +RUN apk add --no-cache \ + chromium \ + nss \ + freetype \ + freetype-dev \ + harfbuzz \ + ca-certificates \ + ttf-freefont + +# Install mermaid CLI globally +RUN npm install -g @mermaid-js/mermaid-cli + +# Set Puppeteer to use the system chromium installation +ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \ + PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser + +# Create app directory +WORKDIR /app + +# Copy the conversion script first (as root) +COPY convert-docs.js /app/convert-docs.js + +# Create package.json and install local dependencies +RUN echo '{"name": "docs-converter", "version": "1.0.0", "dependencies": {"marked": "^11.0.0"}}' > package.json && \ + npm install + +# Create non-root user for security +RUN addgroup -g 1001 -S nodejs +RUN adduser -S nodejs -u 1001 + +# Change ownership of the app directory +RUN chown -R nodejs:nodejs /app +USER nodejs + +# Entry point will be a custom script +ENTRYPOINT ["node", "/app/convert-docs.js"] \ No newline at end of file diff --git a/docs/ai.generated/scripts/README.md b/docs/ai.generated/scripts/README.md new file mode 100644 index 00000000..ee345d24 --- /dev/null +++ b/docs/ai.generated/scripts/README.md @@ -0,0 +1,83 @@ +# PDF Generation Scripts + +This directory contains scripts to generate a professional PDF from the documentation with rendered Mermaid diagrams and working internal links. + +## Files + +- **`Dockerfile`** - Docker container with Mermaid CLI and custom conversion script +- **`convert-docs.js`** - Custom Node.js script that handles Mermaid rendering and link fixing +- **`build-and-generate-pdf.sh`** - macOS/Linux script + +## Usage + +### Prerequisites +- Docker installed and running +- Run from the `docs_temp/scripts` directory + +### macOS/Linux +```bash +cd docs_temp/scripts +./build-and-generate-pdf.sh +``` + +## What the Script Does + +1. **Builds Docker Container** - Creates a container with md-to-pdf and all dependencies +2. **Combines Documentation** - Concatenates all markdown files in proper order +3. **Adds Page Breaks** - Inserts page breaks between major sections +4. **Renders PDF** - Converts markdown to PDF with: + - Rendered Mermaid diagrams + - Professional styling + - Table of contents + - Code syntax highlighting + - Proper margins and formatting + +## Output + +The script generates a timestamped PDF file: +- **Format**: `DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf` +- **Location**: `docs_temp/` directory +- **Features**: + - Working internal links + - Rendered Mermaid diagrams as graphics + - Professional formatting + - Page breaks between sections + +## Troubleshooting + +### Docker Issues +```bash +# Check Docker is running +docker --version + +# Check for container conflicts +docker ps -a | grep doczy-docs-generator +docker rm -f doczy-docs-generator # if needed +``` + +### Permission Issues (Linux/macOS) +```bash +chmod +x build-and-generate-pdf.sh +``` + +### Large File Generation +The PDF may be several MB due to embedded diagrams and styling. This is normal for comprehensive documentation. + +## Customization + +### PDF Options +Edit the `--pdf-options` in the script to customize: +- Page size (A4, Letter, etc.) +- Margins +- Background printing +- Print quality + +### Styling +Modify the `--css-string` section to change: +- Fonts and colors +- Table styling +- Code block appearance +- Header formatting + +### File Order +Change the concatenation order in the script to rearrange sections. \ No newline at end of file diff --git a/docs/ai.generated/scripts/build-and-generate-pdf.sh b/docs/ai.generated/scripts/build-and-generate-pdf.sh new file mode 100755 index 00000000..b55e5ac6 --- /dev/null +++ b/docs/ai.generated/scripts/build-and-generate-pdf.sh @@ -0,0 +1,212 @@ +#!/bin/bash + +# Script to build Docker container and generate PDF documentation +# Run this script from the docs_temp/scripts directory + +set -e # Exit on any error + +echo "🚀 Building Docker container for PDF generation..." + +# Build the Docker image +docker build -t doczy-docs-generator . + +echo "📝 Preparing documentation files..." + +# Go back to docs_temp directory +cd .. + +# Create complete documentation file with proper order and formatting +cat > complete-documentation.md << 'EOF' +# DoczyAI Query Orchestration Platform +## System Documentation + +EOF + +# Concatenate all documentation files in logical order +echo "📚 Combining documentation files..." +cat README.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 01-system-overview.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 02-service-architecture.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 03-api-documentation.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 04-data-architecture.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 05-getting-started.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 06-code-organization.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 07-configuration-management.md >> complete-documentation.md +echo -e "\n\n\\pagebreak\n" >> complete-documentation.md + +cat 08-operations-guide.md >> complete-documentation.md + +echo "🔧 Creating custom CSS file..." + +# Create a custom CSS file for styling +cat > custom-styles.css << 'CSS_EOF' +body { + font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; + line-height: 1.6; + color: #333; + max-width: none; +} + +h1, h2, h3, h4, h5, h6 { + color: #2c3e50; + margin-top: 2em; + margin-bottom: 1em; + page-break-after: avoid; +} + +h1 { + border-bottom: 3px solid #3498db; + padding-bottom: 10px; + page-break-before: always; +} + +h1:first-child { + page-break-before: avoid; +} + +h2 { + border-bottom: 2px solid #ecf0f1; + padding-bottom: 5px; +} + +code { + background-color: #f8f9fa; + padding: 2px 4px; + border-radius: 3px; + font-family: "Monaco", "Menlo", "Ubuntu Mono", monospace; + font-size: 0.9em; +} + +pre { + background-color: #f8f9fa; + padding: 16px; + border-radius: 6px; + border-left: 4px solid #3498db; + overflow-x: auto; + page-break-inside: avoid; +} + +blockquote { + border-left: 4px solid #3498db; + margin: 0; + padding-left: 16px; + color: #6c757d; + font-style: italic; +} + +table { + border-collapse: collapse; + width: 100%; + margin: 1em 0; + page-break-inside: avoid; +} + +th, td { + border: 1px solid #dee2e6; + padding: 8px 12px; + text-align: left; +} + +th { + background-color: #f8f9fa; + font-weight: 600; +} + +.mermaid { + text-align: center; + margin: 2em 0; + page-break-inside: avoid; +} + +a { + color: #3498db; + text-decoration: none; +} + +a:hover { + text-decoration: underline; +} + +/* Page break utilities */ +.pagebreak { + page-break-before: always; +} + +/* Print-specific styles */ +@media print { + body { + font-size: 12pt; + } + + h1 { font-size: 18pt; } + h2 { font-size: 16pt; } + h3 { font-size: 14pt; } + + pre, code { + font-size: 10pt; + } +} + +/* Ensure diagrams and code blocks don't break across pages */ +pre, .mermaid, table { + page-break-inside: avoid; +} +CSS_EOF + +echo "🔧 Skipping shell preprocessing - letting JavaScript handle all link processing..." + +# Don't preprocess links in shell - let the JavaScript handle it properly + +echo "🔧 Generating PDF with Docker container..." + +# Run the Docker container with our custom conversion script and security options for Mermaid rendering +docker run --rm \ + --security-opt seccomp=unconfined \ + --cap-add=SYS_ADMIN \ + -v "$(pwd):/app/docs" \ + -w /app/docs \ + doczy-docs-generator \ + complete-documentation.md + +# Check if PDF was generated successfully +if [ -f "complete-documentation.pdf" ]; then + echo "✅ PDF generated successfully: complete-documentation.pdf" + + # Get file size for confirmation + SIZE=$(ls -lh complete-documentation.pdf | awk '{print $5}') + echo "📄 File size: $SIZE" + + # Move PDF to more descriptive name with timestamp + TIMESTAMP=$(date +"%Y%m%d_%H%M%S") + mv complete-documentation.pdf "DoczyAI-Documentation-${TIMESTAMP}.pdf" + echo "📁 Final file: DoczyAI-Documentation-${TIMESTAMP}.pdf" +else + echo "❌ Error: PDF generation failed" + exit 1 +fi + +# Clean up temporary files +rm -f complete-documentation.md custom-styles.css + +echo "🎉 Documentation PDF generation complete!" +echo "" +echo "The generated PDF includes:" +echo " • Table of contents with working links" +echo " • Rendered Mermaid diagrams" +echo " • Proper page breaks between sections" +echo " • Professional formatting and styling" +echo " • Code syntax highlighting" \ No newline at end of file diff --git a/docs/ai.generated/scripts/convert-docs.js b/docs/ai.generated/scripts/convert-docs.js new file mode 100644 index 00000000..4bef8492 --- /dev/null +++ b/docs/ai.generated/scripts/convert-docs.js @@ -0,0 +1,153 @@ +#!/usr/bin/env node + +const fs = require('fs'); +const path = require('path'); +const { execSync } = require('child_process'); +const { marked } = require('marked'); + +// Custom script to handle Mermaid rendering and fix internal links +async function convertMarkdownToPDF() { + const inputFile = process.argv[2] || 'complete-documentation.md'; + const outputFile = inputFile.replace('.md', '.pdf'); + + console.log(`Converting ${inputFile} to PDF with Mermaid support...`); + + try { + // Read the markdown file + let content = fs.readFileSync(`/app/docs/${inputFile}`, 'utf8'); + + // Remove YAML front matter if present + content = content.replace(/^---\n[\s\S]*?\n---\n/, ''); + + // Remove emojis that don't render well in PDF + content = content.replace(/📋|🛠️|🔧|📊|🏗️|🎉|✅|❌|🚀|📝|📚|🔍|⚠️|💡/g, ''); + + // Helper function to generate section ID from text + function generateSectionId(text) { + return text + .toLowerCase() + .replace(/[^\w\s-]/g, '') // Remove special chars + .replace(/\s+/g, '-') // Replace spaces with hyphens + .replace(/-+/g, '-') // Collapse multiple hyphens + .replace(/^-|-$/g, ''); // Remove leading/trailing hyphens + } + + // Fix internal links to work within the same document + // Convert [text](filename.md) to [text](#section-name) + let linkCount = 0; + content = content.replace(/\[([^\]]+)\]\((\d+-[^)]+\.md)\)/g, (match, linkText, filename) => { + // Use the link text to generate the section ID (since that's what the header will be) + const sectionId = generateSectionId(linkText); + linkCount++; + return `[${linkText}](#${sectionId})`; + }); + + // Fix relative links ./filename.md + content = content.replace(/\[([^\]]+)\]\(\.\/(\d+-[^)]+\.md)\)/g, (match, linkText, filename) => { + const sectionId = generateSectionId(linkText); + linkCount++; + return `[${linkText}](#${sectionId})`; + }); + + console.log(`✅ Processed ${linkCount} internal links`); + + // Process Mermaid diagrams + let mermaidCounter = 0; + content = content.replace(/```mermaid\n([\s\S]*?)\n```/g, (match, mermaidCode) => { + mermaidCounter++; + const diagramFile = `/app/docs/diagram-${mermaidCounter}.mmd`; + const imageFile = `/app/docs/diagram-${mermaidCounter}.png`; + + // Write mermaid code to temporary file + fs.writeFileSync(diagramFile, mermaidCode.trim()); + + try { + // Generate PNG from mermaid + execSync(`mmdc -i ${diagramFile} -o ${imageFile} -w 800 -H 600 --backgroundColor white`); + + // Replace with image reference + return `![Diagram ${mermaidCounter}](diagram-${mermaidCounter}.png)`; + } catch (error) { + console.warn(`Warning: Failed to render Mermaid diagram ${mermaidCounter}:`, error.message); + // Return original code block if mermaid fails + return match; + } + }); + + // Configure marked to generate proper IDs for headers + marked.use({ + renderer: { + heading(text, level) { + // Generate ID from text content + const id = text + .toLowerCase() + .replace(/[^\w\s-]/g, '') // Remove special chars + .replace(/\s+/g, '-') // Replace spaces with hyphens + .replace(/-+/g, '-') // Collapse multiple hyphens + .replace(/^-|-$/g, ''); // Remove leading/trailing hyphens + + return `${text}`; + } + } + }); + + // Convert markdown to HTML + const html = marked(content, { + gfm: true, + breaks: false, + headerIds: true, + mangle: false + }); + + // Create complete HTML document with CSS + const cssContent = fs.readFileSync('/app/docs/custom-styles.css', 'utf8'); + const fullHtml = ` + + + + + DoczyAI Documentation + + + + ${html} + +`; + + // Write HTML to temporary file + const htmlFile = `/app/docs/temp-${inputFile.replace('.md', '.html')}`; + fs.writeFileSync(htmlFile, fullHtml); + + // Use Chromium to generate PDF with clickable links (suppress stderr) + execSync(`/usr/bin/chromium-browser --headless --disable-gpu --no-sandbox --run-all-compositor-stages-before-draw --virtual-time-budget=10000 --print-to-pdf=/app/docs/${outputFile} --print-to-pdf-no-header file://${htmlFile} 2>/dev/null`); + + console.log(`✅ PDF generated: ${outputFile}`); + + // Clean up temporary files + try { + fs.unlinkSync(htmlFile); + for (let i = 1; i <= mermaidCounter; i++) { + try { + fs.unlinkSync(`/app/docs/diagram-${i}.mmd`); + // Keep PNG files for debugging, remove in production + // fs.unlinkSync(`/app/docs/diagram-${i}.png`); + } catch (e) { + // Ignore cleanup errors + } + } + } catch (e) { + // Ignore cleanup errors + } + + } catch (error) { + console.error('❌ Error converting to PDF:', error.message); + console.error(error.stack); + process.exit(1); + } +} + +convertMarkdownToPDF(); \ No newline at end of file diff --git a/docs/ai.generated/scripts/mermaid-to-url.sh b/docs/ai.generated/scripts/mermaid-to-url.sh new file mode 100755 index 00000000..2b120fde --- /dev/null +++ b/docs/ai.generated/scripts/mermaid-to-url.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +# Script to convert piped mermaid text to a mermaid.live URL +# Usage: cat mermaid.md | ./mermaid-to-url.sh +# or: echo "graph TD; A-->B" | ./mermaid-to-url.sh + +# This script delegates to the Python version for proper compression +# Get the directory where this script is located +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Call the Python script +python3 "$SCRIPT_DIR/mermaid-to-url.py" \ No newline at end of file diff --git a/docs/ai.generated/service.details.md b/docs/ai.generated/service.details.md new file mode 100644 index 00000000..056530f8 --- /dev/null +++ b/docs/ai.generated/service.details.md @@ -0,0 +1,362 @@ +# Service Processing Details + +This document provides detailed information about what each service in the document processing pipeline actually does. + +## Document Processing Pipeline Services + +### 1. storeEventRunner +**Purpose**: Entry point for document processing, triggered by S3 upload events + +**Code Locations**: +- `cmd/storeEventRunner/` - Service entry point and main.go +- `api/storeEventRunner/` - Runner implementation and message handling +- `internal/document/store/` - Core document storage logic +- `internal/serviceconfig/queue/documentinit/` - Queue configuration for next stage + +**Processing Logic**: +- Receives S3 event notifications (ObjectCreated:Put, ObjectCreated:CompleteMultipartUpload) +- Validates that the event type is supported +- Parses the S3 object key to extract structured information (client ID, location, timestamps) +- Filters for objects in the "import" location only +- Extracts document metadata (bucket, key, hash/ETag) +- Sends message to DOCINIT queue with document location information + +**Key Validations**: +- Only processes supported S3 events (Put and CompleteMultipartUpload) +- Only processes objects with "import" location in their key path +- Validates key can be parsed into expected structure + +**Output**: Sends to DOCINIT queue with {bucket, key, hash} + +--- + +### 2. docInitRunner +**Purpose**: Document registration and deduplication + +**Code Locations**: +- `cmd/docInitRunner/` - Service entry point and main.go +- `api/docInitRunner/` - Runner implementation and message handling +- `internal/document/init/` - Core document initialization logic +- `internal/serviceconfig/queue/documentsync/` - Queue configuration for next stage + +**Processing Logic**: +- Parses the S3 key to extract client ID, batch ID, and other metadata +- Checks if document already exists by looking up hash in database for the client +- If document exists (duplicate): + - Adds new document entry to existing document record + - Does NOT trigger downstream processing (deduplication) +- If new document: + - Creates new document record in database with client ID and hash + - Associates batch ID if present + - Adds document entry with S3 location details + - Sends message to DOCSYNC queue to continue processing + +**Key Features**: +- Hash-based deduplication per client +- Maintains multiple entries for same document (different uploads) +- Transaction-based database operations for consistency + +**Output**: Sends to DOCSYNC queue with {document_id} (only for new documents) + +--- + +### 3. docSyncRunner +**Purpose**: Client synchronization validation + +**Code Locations**: +- `cmd/docSyncRunner/` - Service entry point and main.go +- `api/docSyncRunner/` - Runner implementation and message handling +- `internal/document/sync/` - Core document synchronization logic +- `internal/client/` - Client management utilities +- `internal/document/` - Shared document utilities +- `internal/serviceconfig/queue/documentclean/` - Queue configuration for next stage + +**Processing Logic**: +- Retrieves document summary from database +- Fetches client configuration +- Checks if client has "CanSync" flag enabled +- If CanSync is true: + - Sends document to DOCCLEAN queue for processing +- If CanSync is false: + - Logs that document won't be synced + - Processing stops here for this document + +**Key Validations**: +- Client must exist in database +- Client must have CanSync=true to proceed + +**Output**: Sends to DOCCLEAN queue with {document_id} (only if client can sync) + +--- + +### 4. docCleanRunner +**Purpose**: Document validation and preparation + +**Code Locations**: +- `cmd/docCleanRunner/` - Service entry point and main.go +- `api/docCleanRunner/` - Runner implementation and message handling +- `internal/document/clean/` - Core document cleaning and validation logic +- `internal/document/types/` - Document type definitions and MIME type handling +- `internal/serviceconfig/queue/documenttext/` - Queue configuration for next stage + +**Processing Logic**: +- Checks if document has already been cleaned (idempotency) +- If not cleaned: + - Retrieves document entry from database + - Makes HEAD request to S3 to get content metadata + - Validates content type/MIME type is acceptable (PDF) + - Downloads document from S3 + - Validates document integrity (not corrupt) + - Stores clean record with: + - Success: bucket, key, hash, mimetype + - Failure: reason for failure (invalid MIME, corrupt file) +- Creates versioned clean entry for tracking +- Sends to DOCTEXT queue regardless of success/failure + +**Key Validations**: +- MIME type validation (must be acceptable format) +- Document corruption check +- Hash verification against S3 ETag + +**Output**: Sends to DOCTEXT queue with {document_id} + +--- + +### 5. docTextRunner +**Purpose**: Text extraction from documents using AWS Textract + +**Code Locations**: +- `cmd/docTextRunner/` - Service entry point and main.go +- `api/docTextRunner/` - Runner implementation and message handling +- `internal/document/text/` - Core text extraction logic using AWS Textract +- `internal/document/types/` - Document type utilities for text processing +- `internal/serviceconfig/textract/` - AWS Textract client configuration +- `internal/serviceconfig/queue/querysync/` - Queue configuration for next stage + +Tests for this use mocks only. No tests to the actual aws textract service. + +**Processing Logic**: +- Checks if text has already been extracted (idempotency) +- If not extracted: + - Retrieves clean document record + - Skips if document failed cleaning + - Downloads document from S3 + - Determines page count + - For each page (in parallel using thread pool): + - Sends page to AWS Textract for OCR + - Processes base text blocks + - Identifies required features (tables, forms) + - Performs additional extraction if needed + - Merges text with page markers + - Stores extracted text in S3 + - Records text extraction in database with versioning +- Sends to QUERYSYNC queue for query processing + +**Key Features**: +- Parallel page processing for performance +- Supports tables and forms extraction +- Stores full text with page boundaries preserved +- Version tracking for text extractions + +**Output**: Sends to QUERYSYNC queue with {document_id} + +--- + +### 6. querySyncRunner +**Purpose**: Query dependency resolution and scheduling + +**Code Locations**: +- `cmd/querySyncRunner/` - Service entry point and main.go +- `api/querySyncRunner/` - Runner implementation and message handling +- `internal/query/sync/` - Core query synchronization logic +- `internal/query/result/sync/` - Result synchronization utilities +- `internal/serviceconfig/queue/query/` - Queue configuration for next stage + +**Processing Logic**: +- Verifies document has extracted text +- Queries database for unsynced queries that: + - Are configured for this client (via collector) + - Have no dependencies OR + - Have all dependencies already satisfied +- For each eligible query: + - Sends message to QUERY queue for execution + +**Key Features**: +- Dependency graph resolution +- Only schedules queries with satisfied dependencies +- Client-specific query configuration via collectors + +**Output**: Sends to QUERY queue with {document_id, query_id} for each eligible query + +--- + +### 7. queryRunner +**Purpose**: Query execution and result storage + +**Code Locations**: +- `cmd/queryRunner/` - Service entry point and main.go +- `api/queryRunner/` - Runner implementation and message handling +- `internal/query/result/set/` - Core query result processing +- `internal/query/result/` - Query execution logic and processors +- `internal/query/result/processor/` - Base processor interface and utilities +- `internal/query/types/contextFull/` - CONTEXT_FULL query type implementation +- `internal/query/types/jsonExtractor/` - JSON_EXTRACTOR query type implementation +- `internal/query/` - Query management utilities +- `internal/query/result/sync/` - Result synchronization for triggering dependents + +**Processing Logic**: +- Retrieves document text version information +- Fetches query definition and active version +- If query has dependencies: + - Retrieves results from required queries + - Validates all dependencies are satisfied +- Executes query based on type: + - **CONTEXT_FULL**: Returns complete document text + - **JSON_EXTRACTOR**: Extracts specific JSON paths from dependent results +- Stores result in database with: + - Query result value + - Query version used + - Text version processed + - Dependencies tracked +- Identifies queries that depend on this result +- Sends dependent queries to QUERY queue (recursive processing) + +**Key Features**: +- Supports multiple query types with different processors +- Dependency tracking and validation +- Recursive processing for dependent queries +- Version tracking for reproducibility + +**Output**: Sends to QUERY queue with {document_id, query_id} for each dependent query + +--- + +## Additional Synchronization Services + +### 8. clientSyncRunner +**Purpose**: Client-level document synchronization + +**Code Locations**: +- `cmd/clientSyncRunner/` - Service entry point and main.go +- `api/clientSyncRunner/` - Runner implementation and message handling +- `internal/client/sync/` - Core client synchronization logic +- `internal/serviceconfig/queue/documentsync/` - Queue configuration for document sync + +**Processing Logic**: +- Processes CLIENTSYNC queue messages +- Retrieves all documents for a specific client +- Sends each document to DOCSYNC queue for processing +- Enables bulk reprocessing of client documents + +**Output**: Sends to DOCSYNC queue with {document_id} for each client document + +--- + +### 9. queryVersionSyncRunner +**Purpose**: Query version change propagation + +**Code Locations**: +- `cmd/queryVersionSyncRunner/` - Service entry point and main.go +- `api/queryVersionSyncRunner/` - Runner implementation and message handling +- `internal/query/versionsync/` - Core query version synchronization logic +- `internal/serviceconfig/queue/clientsync/` - Queue configuration for client sync + +**Processing Logic**: +- Processes QUERYVERSIONSYNC queue messages triggered by query updates +- Identifies all clients affected by query version changes +- Sends affected clients to CLIENTSYNC queue for reprocessing +- Enables cascading updates when query logic changes + +**Output**: Sends to CLIENTSYNC queue with {client_id} for each affected client + +--- + +## Data Flow Summary + +### Primary Document Processing Pipeline +``` +S3 Upload Event + ↓ +storeEventRunner (validates event, extracts metadata) + ↓ +docInitRunner (deduplication, document registration) + ↓ +docSyncRunner (client sync validation) + ↓ +docCleanRunner (document validation, MIME check) + ↓ +docTextRunner (OCR via AWS Textract) + ↓ +querySyncRunner (find executable queries) + ↓ +queryRunner (execute queries, trigger dependents) + ↓ +queryRunner (recursive for dependent queries) +``` + +### Synchronization Flows +``` +Query Version Update + ↓ +queryVersionSyncRunner (find affected clients) + ↓ +clientSyncRunner (reprocess all client documents) + ↓ +docSyncRunner → ... (normal processing pipeline) +``` + +## Key Design Patterns + +### Idempotency +- Each service checks if its work has already been done +- Prevents duplicate processing on retries +- Version tracking ensures consistency + +### Dependency Management +- Queries can depend on other query results +- Dependency graph is resolved before execution +- Results are chained through recursive processing + +### Version Tracking +- Document clean versions +- Text extraction versions +- Query versions +- Enables reprocessing when configurations change + +### Error Handling +- Failed documents are marked but continue through pipeline +- Allows partial processing and debugging +- Failure reasons are stored for analysis + +### Client Isolation +- All processing is scoped to client ID +- Deduplication is per-client +- Query configurations are per-client via collectors + +--- + +## Shared Code Directories + +These directories contain utilities and configurations used across multiple services: + +### Core Infrastructure +- `internal/database/` - Database queries, migrations, and repository patterns +- `internal/serviceconfig/` - Configuration management for all services +- `internal/serviceconfig/queue/` - SQS queue client configurations +- `internal/serviceconfig/objectstore/` - S3 client configurations +- `internal/serviceconfig/aws/` - AWS service configurations +- `internal/server/runner/` - Base runner framework for queue-based services + +### Business Logic Libraries +- `internal/document/` - General document utilities (shared across document services) +- `internal/client/` - Client management utilities +- `internal/query/` - Query management and utilities (shared across query services) + +### Testing & Utilities +- `internal/test/` - Test utilities and helpers +- `internal/validation/` - Input validation utilities +- `mocks/` - Auto-generated mock interfaces for testing + +### API Layer +- `api/queryAPI/` - REST API handlers and controllers (not a queue-based runner) +- `pkg/queryAPI/` - Generated API client libraries \ No newline at end of file