Merged in feature/ai-generated-docs (pull request #167)
ai generated docs and related tools for publishing new versions * ai generated docs and tooling * tools * add mermaid links * add task docs:ai:generate * Merged main into feature/ai-generated-docs Approved-by: Michael McGuinness
This commit is contained in:
Jay Brown
@@ -0,0 +1,136 @@
|
||||
# System Overview
|
||||
|
||||
## DoczyAI Query Orchestration Platform
|
||||
|
||||
### Purpose and Domain
|
||||
DoczyAI is a cloud-native document processing and query orchestration platform designed to extract structured information from documents (primarily PDFs) using configurable, hierarchical queries. The system serves as a multi-tenant platform where organizations can upload documents, define custom data extraction rules, and receive structured results for business intelligence and analysis.
|
||||
|
||||
### Core Business Capabilities
|
||||
- **Multi-tenant Document Processing**: Isolated processing environments per client
|
||||
- **Configurable Query System**: Custom data extraction rules with dependency management
|
||||
- **Automated Processing Pipeline**: Seamless flow from document upload to structured output
|
||||
- **Version Management**: Complete versioning for queries and configurations
|
||||
- **Export and Integration**: Structured data export capabilities for downstream systems
|
||||
|
||||
### Technology Stack
|
||||
- **Language**: Go 1.24.0
|
||||
- **Web Framework**: Echo v4 with OpenAPI code generation
|
||||
- **Database**: PostgreSQL 17.2 with SQLC for type-safe queries
|
||||
- **Messaging**: AWS SQS for event-driven microservices
|
||||
- **Storage**: AWS S3 for document storage
|
||||
- **Text Processing**: AWS Textract for OCR and text extraction
|
||||
- **Authentication**: AWS Cognito with JWT-based authorization
|
||||
- **Monitoring**: Prometheus metrics and OpenTelemetry tracing
|
||||
- **Container Platform**: Docker with multi-stage builds
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### System Architecture Pattern
|
||||
The system follows an **event-driven microservices architecture** with the following characteristics:
|
||||
- **12 distinct services**: 1 HTTP API + 8 queue-based runners + 3 utility services
|
||||
- **Asynchronous communication**: SQS queues between services
|
||||
- **State management**: PostgreSQL as central data store
|
||||
- **External service integration**: AWS managed services for cloud capabilities
|
||||
|
||||
### High-Level Component Diagram
|
||||
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNqFlGtr2zAUhv-K8KcOVtYtG0vHKCSyKYa2sSM3HTilKM5JYnCkTpe2Wel_ny4OlZ1C_MGWjp5z0asjv0YVX0L0C0VrQR83qBjPGTKP1AtvmEfJiwLBaINwUwNTch55xD63aXkHC5Qyg6xoBffvS6MsfcBXaXJTlGa4d24BYEs_OMhm2Uuq4JnuwkS5sZd_NYidGfxeiC8XGRcKDc-GZ0dDxrzSW5McZYJXIGXN1iirH6GpGYQ5SFJKxQUkTwaeasZAuEwmydd7dHp6geK0XPIqZXVv-Vuw7zj1KLEo2bGqiw5ClHgUWxQ3QFmX_R6y2LOFZQt46VXwI0QLh-bEC3ZYw88Azn0N-dTDXXB4VFsbfCM4q_9RVXOGCIin2ogc6opJWbnTP6zkPKxk1hY8AyFNrAP6_PhRp2wlqFRCV0oL-LCaeFyeZFyqtQCSX7nQMVV0QSV8Cqohg_JkdEfM1yHENAZddwkjsCNy4pBcgwYZAEXyp5iOcOEoe2SCVsqhdoISbzA7DXzw5PImLSbOBfO16TTuPEZabYyCdUV7Dtl0cl2axt6CAbR08DW3fsL0-ceC3foWtXfKG97vam_Bv-3cd9-4ZyCDviEnPUu7ozAeSbps3I2-vxb7Ke6kart7L26breuSTztT_8akV-GsZ2i1aJqHfd-4datw9BlFWxBbWi_Nv_I1Mlpv3V9zCSuqGxW9vf0HHgWInQ==)
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "External Clients"
|
||||
UI[Web Interface]
|
||||
API_CLIENT[API Clients]
|
||||
end
|
||||
|
||||
subgraph "API Gateway"
|
||||
QAPI[queryAPI<br/>Port 8080]
|
||||
end
|
||||
|
||||
subgraph "Document Processing Pipeline"
|
||||
SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
|
||||
DI --> DS[docSyncRunner<br/>8083]
|
||||
DS --> DC[docCleanRunner<br/>8084]
|
||||
DC --> DT[docTextRunner<br/>8085]
|
||||
DT --> QS[querySyncRunner<br/>8087]
|
||||
QS --> QR[queryRunner<br/>8088]
|
||||
end
|
||||
|
||||
subgraph "Synchronization Services"
|
||||
CS[clientSyncRunner<br/>8089]
|
||||
QVS[queryVersionSyncRunner<br/>8090]
|
||||
end
|
||||
|
||||
subgraph "Infrastructure Services"
|
||||
DB[(PostgreSQL<br/>Database)]
|
||||
S3[(AWS S3<br/>Storage)]
|
||||
SQS[AWS SQS<br/>Queues]
|
||||
TEXTRACT[AWS Textract<br/>Text Extraction]
|
||||
COGNITO[AWS Cognito<br/>Authentication]
|
||||
PROM[Prometheus<br/>Monitoring]
|
||||
end
|
||||
|
||||
UI --> QAPI
|
||||
API_CLIENT --> QAPI
|
||||
|
||||
QAPI --> DB
|
||||
QAPI --> S3
|
||||
QAPI --> SQS
|
||||
QAPI --> COGNITO
|
||||
|
||||
SE --> SQS
|
||||
DI --> DB
|
||||
DS --> DB
|
||||
DC --> S3
|
||||
DT --> TEXTRACT
|
||||
QS --> DB
|
||||
QR --> DB
|
||||
|
||||
CS --> SQS
|
||||
QVS --> SQS
|
||||
|
||||
All_Services --> PROM
|
||||
```
|
||||
|
||||
## Deployment Architecture
|
||||
|
||||
### Infrastructure Requirements
|
||||
- **Compute**: Container orchestration platform (Docker/Kubernetes)
|
||||
- **Database**: PostgreSQL 17.2+ with connection pooling
|
||||
- **AWS Services**: S3, SQS, Textract, Cognito services
|
||||
- **Monitoring**: Prometheus-compatible metrics collection
|
||||
- **Networking**: Load balancer for API endpoints
|
||||
|
||||
### Scalability Considerations
|
||||
- **Horizontal Scaling**: All runners can be scaled independently based on queue depth
|
||||
- **Database Scaling**: Connection pooling with configurable pool sizes
|
||||
- **Storage Scaling**: S3 provides virtually unlimited document storage
|
||||
- **Processing Scaling**: Auto-scaling based on SQS queue metrics
|
||||
|
||||
### Security Architecture
|
||||
- **Authentication**: OAuth2 with AWS Cognito integration
|
||||
- **Authorization**: Role-based access control with group-based permissions
|
||||
- **Data Protection**: Encrypted storage (S3 server-side encryption)
|
||||
- **Network Security**: VPC isolation and security groups
|
||||
- **API Security**: JWT validation and OpenAPI request validation
|
||||
|
||||
## Integration Points
|
||||
|
||||
### External System Dependencies
|
||||
1. **AWS S3**: Document storage and retrieval
|
||||
2. **AWS SQS**: Inter-service messaging and event processing
|
||||
3. **AWS Textract**: OCR and text extraction from documents
|
||||
4. **AWS Cognito**: User authentication and session management
|
||||
5. **PostgreSQL**: Central data persistence and query processing
|
||||
|
||||
### API Integration
|
||||
- **REST API**: OpenAPI 3.0.3 specification with Swagger UI
|
||||
- **Authentication**: Bearer token or OAuth2 flows
|
||||
- **Rate Limiting**: 100 requests per 60-second window
|
||||
- **Content Types**: JSON for API, multipart/form-data for uploads
|
||||
|
||||
### Performance Characteristics
|
||||
- **Throughput**: Designed for high-volume document processing
|
||||
- **Latency**: Sub-second API response times for most operations
|
||||
- **Availability**: Distributed architecture supports high availability
|
||||
- **Durability**: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively
|
||||
@@ -0,0 +1,210 @@
|
||||
# Service Architecture
|
||||
|
||||
## Service Catalog
|
||||
|
||||
### Primary Services
|
||||
|
||||
#### queryAPI - Main HTTP API Service
|
||||
- **Purpose**: Central management API for all user entities and operations
|
||||
- **Port**: 8080
|
||||
- **Type**: HTTP REST API
|
||||
- **Key Responsibilities**:
|
||||
- Client management and authentication
|
||||
- Document upload and metadata management
|
||||
- Query definition and testing
|
||||
- Export operations and status monitoring
|
||||
- User interface and API documentation
|
||||
|
||||
#### storeEventRunner - S3 Event Processor
|
||||
- **Purpose**: Entry point for document processing pipeline
|
||||
- **Port**: 8081
|
||||
- **Type**: Event-driven queue consumer
|
||||
- **Input**: S3 PUT event notifications
|
||||
- **Output**: Events to DOCINIT queue
|
||||
- **Responsibilities**: Processes S3 upload events and initiates document processing
|
||||
|
||||
#### docInitRunner - Document Initialization
|
||||
- **Purpose**: Document registration and duplicate detection
|
||||
- **Port**: 8082
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{bucket, key, hash}` from DOCINIT queue
|
||||
- **Output**: Events to DOCSYNC queue
|
||||
- **Key Functions**:
|
||||
- ETag-based duplicate detection
|
||||
- Document reference entity creation
|
||||
- Upload success logging
|
||||
|
||||
#### docSyncRunner - Document Sync Validation
|
||||
- **Purpose**: Validates document eligibility for processing
|
||||
- **Port**: 8083
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from DOCSYNC queue
|
||||
- **Output**: Events to DOCCLEAN queue
|
||||
- **Logic**: Validates client `can_sync` parameter
|
||||
|
||||
#### docCleanRunner - Document Preparation
|
||||
- **Purpose**: Document cleaning and validation
|
||||
- **Port**: 8084
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from DOCCLEAN queue
|
||||
- **Output**: Events to DOCTEXT queue
|
||||
- **Dependencies**: S3 ObjectStore for document access
|
||||
|
||||
#### docTextRunner - Text Extraction
|
||||
- **Purpose**: OCR and text extraction using AWS Textract
|
||||
- **Port**: 8085
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from DOCTEXT queue
|
||||
- **Output**: Events to QUERYSYNC queue
|
||||
- **Dependencies**: AWS Textract, S3 ObjectStore
|
||||
|
||||
#### querySyncRunner - Query Dependency Resolution
|
||||
- **Purpose**: Identifies executable queries for documents
|
||||
- **Port**: 8087
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from QUERYSYNC queue
|
||||
- **Output**: Events to QUERY queue
|
||||
- **Logic**: Finds queries with satisfied dependencies
|
||||
|
||||
#### queryRunner - Query Execution Engine
|
||||
- **Purpose**: Executes queries and stores results
|
||||
- **Port**: 8088
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id, query_id}` from QUERY queue
|
||||
- **Output**: Events to downstream QUERY queue
|
||||
- **Responsibilities**: Query execution and result persistence
|
||||
|
||||
### Synchronization Services
|
||||
|
||||
#### clientSyncRunner - Client-Level Synchronization
|
||||
- **Purpose**: Synchronizes all documents for a client
|
||||
- **Port**: 8089
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{client_id}` from CLIENTSYNC queue
|
||||
- **Output**: Events to DOCSYNC queue for each client document
|
||||
|
||||
#### queryVersionSyncRunner - Query Change Propagation
|
||||
- **Purpose**: Propagates query updates to affected clients
|
||||
- **Port**: 8090
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{query_id}` from QUERYVERSIONSYNC queue
|
||||
- **Output**: Events to CLIENTSYNC queue for affected clients
|
||||
|
||||
### Utility Services
|
||||
|
||||
#### healthcheck - Health Monitoring
|
||||
- **Purpose**: Service health validation
|
||||
- **Type**: HTTP health check utility
|
||||
- **Target**: localhost:8080 health endpoint
|
||||
|
||||
#### textExtractor - Development Utility
|
||||
- **Purpose**: Local text extraction for testing
|
||||
- **Type**: Standalone utility
|
||||
- **Use Case**: Development and testing workflows
|
||||
|
||||
#### metricsExample_test - Monitoring Example
|
||||
- **Purpose**: Demonstrates Prometheus metrics implementation
|
||||
- **Type**: Educational/testing service
|
||||
|
||||
## Data Flow Architecture
|
||||
|
||||
### Document Processing Pipeline
|
||||
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNp9k8FPgzAUxv-VhoMnPe3GYckGmCwxKLCpS7zU8rY1QottMVuW_e_2lWKGiBzIS78f3-v7Ss8BkyUEIQk0fLYgGMSc7hWt3wSxT0OV4Yw3VBgSVRyEGa8Xsz_WjFRAqCYai-TLfpi3QoAaoyvBDZKlZFhOYcVJMI9hOYVFFVDhOVdPgWs49m2xnMKyvq8NR53-65wh8ENOUfESkZga-k41dHr37uK9m8-LWUg2TSVpSWLJ2von9GKGKgYa2pq4VEkqDd9xRg2XwmNIWBLTDEn8GK3S1Rq31_p-KFg9XoYkOgD7IHHbVGgBegDgtM6g2KbRtQEK3uCZVry0X_rtO2kAuUNwLtFDskivbZzkB-4nJU9KMtCai_0QwkNyNuvkdTANCt4kh1FmXl28FKGrFWXjHh7KuoGzTZJvf4-cXc18z0VJkiOw1p5iBe7keZ9dD7rfwZsNjHDdG3UeQG78fclBt5XRwS0JalA15aW9mOfAHKB2V7SEHbVAcLl8AzPQPDE=)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant S3
|
||||
participant Store as storeEventRunner
|
||||
participant Init as docInitRunner
|
||||
participant Sync as docSyncRunner
|
||||
participant Clean as docCleanRunner
|
||||
participant Text as docTextRunner
|
||||
participant QSync as querySyncRunner
|
||||
participant Query as queryRunner
|
||||
participant DB as Database
|
||||
|
||||
Client->>S3: Upload Document
|
||||
S3->>Store: S3 Event Notification
|
||||
Store->>Init: DOCINIT Queue
|
||||
Init->>DB: Check Duplicates
|
||||
Init->>Sync: DOCSYNC Queue
|
||||
Sync->>DB: Validate Client Sync
|
||||
Sync->>Clean: DOCCLEAN Queue
|
||||
Clean->>S3: Document Processing
|
||||
Clean->>Text: DOCTEXT Queue
|
||||
Text->>S3: Read Document
|
||||
Text->>AWS: Textract Processing
|
||||
Text->>QSync: QUERYSYNC Queue
|
||||
QSync->>DB: Find Executable Queries
|
||||
QSync->>Query: QUERY Queue
|
||||
Query->>DB: Execute & Store Results
|
||||
```
|
||||
|
||||
### Query Version Management Flow
|
||||
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNp9kctqwzAQRX9l0KqF9ge8CBTZC0Nx_GgMgW6ENG4GLFmVZEII-ffKj_SVUi2EGJ0z3GHOTA4KWQLM4_uIRmJK4s0J_WogHitcIElWmABPSpP5o1zmIDxE2Z3i-xao2uYTaNF5GkxzMrIejUF3i_OZlj2hCf9xJVnsyeBEp4McNX4rLvhyz7kfN5sYLoGdVSIgVFMYSLEjQyEGWskyj1zMm0C1y-p9m9VNvi2afcEnY1zbRiBiPFL8Oc-Kl69_uLPoQHQdyoBqHeJ-sfgkXfMlkG75b2_BQa3DrN5V-WHXaN0g0Xs4UjhAgcdlJPYATKPTglTc6ZmFA-p5uwo7MfaBXS4faYamhw==)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Admin
|
||||
participant API as queryAPI
|
||||
participant QVS as queryVersionSyncRunner
|
||||
participant CS as clientSyncRunner
|
||||
participant Pipeline as Document Pipeline
|
||||
|
||||
Admin->>API: Update Query Definition
|
||||
API->>QVS: QUERYVERSIONSYNC Queue
|
||||
QVS->>CS: CLIENTSYNC Queue (per affected client)
|
||||
CS->>Pipeline: DOCSYNC Queue (per client document)
|
||||
Pipeline->>Pipeline: Reprocess with New Query
|
||||
```
|
||||
|
||||
## Communication Patterns
|
||||
|
||||
### Inter-Service Messaging
|
||||
All services communicate through **AWS SQS queues** with the following patterns:
|
||||
- **Fire-and-forget**: Services publish events without waiting for responses
|
||||
- **Queue-based**: Loose coupling between services
|
||||
- **Retry logic**: Built-in message retry and dead letter queue support
|
||||
- **Ordering**: FIFO queues where order matters
|
||||
|
||||
### Queue Architecture
|
||||
|
||||
| Queue Name | Producer | Consumer | Message Format |
|
||||
|------------|----------|----------|----------------|
|
||||
| DOCINIT | storeEventRunner | docInitRunner | `{bucket, key, hash}` |
|
||||
| DOCSYNC | docInitRunner, clientSyncRunner | docSyncRunner | `{document_id}` |
|
||||
| DOCCLEAN | docSyncRunner | docCleanRunner | `{document_id}` |
|
||||
| DOCTEXT | docCleanRunner | docTextRunner | `{document_id}` |
|
||||
| QUERYSYNC | docTextRunner | querySyncRunner | `{document_id}` |
|
||||
| QUERY | querySyncRunner, queryRunner | queryRunner | `{document_id, query_id}` |
|
||||
| CLIENTSYNC | queryVersionSyncRunner | clientSyncRunner | `{client_id}` |
|
||||
| QUERYVERSIONSYNC | queryAPI | queryVersionSyncRunner | `{query_id}` |
|
||||
|
||||
### Database Access Patterns
|
||||
- **Repository Pattern**: SQLC-generated type-safe database access
|
||||
- **Connection Pooling**: Shared database connections across services
|
||||
- **Transaction Management**: ACID transactions for data consistency
|
||||
- **Read/Write Separation**: Potential for read replica scaling
|
||||
|
||||
## Scalability and Performance
|
||||
|
||||
### Horizontal Scaling Strategy
|
||||
- **API Service**: Load balancer with multiple instances
|
||||
- **Queue Processors**: Auto-scaling based on queue depth
|
||||
- **Database**: Read replicas for query scaling
|
||||
- **Storage**: S3 scales automatically
|
||||
|
||||
### Performance Optimizations
|
||||
- **Connection Pooling**: Database connection reuse
|
||||
- **Batch Processing**: Queue messages processed in batches
|
||||
- **Caching**: JWKS caching for authentication
|
||||
- **Compression**: Gzip compression for API responses
|
||||
|
||||
### Monitoring and Observability
|
||||
- **Metrics**: Prometheus metrics from all services
|
||||
- **Tracing**: OpenTelemetry distributed tracing
|
||||
- **Logging**: Structured logging with correlation IDs
|
||||
- **Health Checks**: Built-in health endpoints
|
||||
@@ -0,0 +1,345 @@
|
||||
# API Documentation
|
||||
|
||||
## API Overview
|
||||
|
||||
The DoczyAI Query Orchestration Platform exposes a comprehensive REST API following OpenAPI 3.0.3 specification. The API is automatically generated from the specification file `serviceAPIs/queryAPI.yaml` using oapi-codegen.
|
||||
|
||||
### Base Configuration
|
||||
- **Base URL**: `https://doczy.com/v1` (production), `http://localhost:8080` (development)
|
||||
- **API Version**: 0.0.1
|
||||
- **Content Type**: `application/json`
|
||||
- **Documentation**: Auto-generated Swagger UI at `/swagger/index.html`
|
||||
|
||||
## Authentication and Authorization
|
||||
|
||||
### Security Schemes
|
||||
|
||||
#### JWT Authentication
|
||||
- **Type**: Bearer token authentication
|
||||
- **Header**: `Authorization: Bearer <token>`
|
||||
- **Token Source**: JWT cookie set via OAuth flow
|
||||
|
||||
#### OAuth2 Flow
|
||||
- **Type**: OAuth2 Authorization Code Flow
|
||||
- **Provider**: AWS Cognito
|
||||
- **Scopes**: Group-based permissions
|
||||
|
||||
### Authentication Endpoints
|
||||
|
||||
#### `GET /login`
|
||||
Initiates OAuth2 authentication flow with AWS Cognito.
|
||||
|
||||
**Response**: Redirect to Cognito login page
|
||||
|
||||
#### `GET /login-callback`
|
||||
OAuth2 callback handler that processes authentication response.
|
||||
|
||||
**Parameters**:
|
||||
- `code` (query): Authorization code from Cognito
|
||||
- `state` (query): CSRF protection parameter
|
||||
|
||||
**Response**: Sets JWT cookie and redirects to dashboard
|
||||
|
||||
#### `GET /logout`
|
||||
Terminates user session and clears authentication cookies.
|
||||
|
||||
**Response**: Redirect to login page
|
||||
|
||||
#### `GET /home`
|
||||
Protected dashboard endpoint showing user information.
|
||||
|
||||
**Security**: Requires authentication
|
||||
**Response**: HTML dashboard page
|
||||
|
||||
### Authorization Model
|
||||
|
||||
#### Permission Groups
|
||||
- **exporters**: Can access export operations
|
||||
- **uploaders**: Can upload documents and manage clients
|
||||
- **querybuilders**: Can create and modify queries
|
||||
|
||||
#### Route Permissions
|
||||
```
|
||||
/client/* → exporters, uploaders, querybuilders
|
||||
/document/* → exporters, querybuilders
|
||||
/query/* → exporters, uploaders, querybuilders
|
||||
/export/* → exporters only
|
||||
/health → public (no authentication required)
|
||||
```
|
||||
|
||||
## Core API Endpoints
|
||||
|
||||
### Client Management
|
||||
|
||||
#### `POST /client`
|
||||
Creates a new client in the system.
|
||||
|
||||
**Security**: Requires `uploaders` or `querybuilders` permission
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"clientId": "string (max 256 chars)",
|
||||
"name": "string (max 256 chars)"
|
||||
}
|
||||
```
|
||||
|
||||
**Responses**:
|
||||
- `201`: Client created successfully
|
||||
- `400`: Invalid request data
|
||||
- `401`: Authentication required
|
||||
- `403`: Insufficient permissions
|
||||
|
||||
#### `GET /client/{id}`
|
||||
Retrieves client details by ID.
|
||||
|
||||
**Path Parameters**:
|
||||
- `id`: Client identifier (string, max 256 chars)
|
||||
|
||||
**Response Example**:
|
||||
```json
|
||||
{
|
||||
"clientId": "acme-corp",
|
||||
"name": "ACME Corporation"
|
||||
}
|
||||
```
|
||||
|
||||
#### `PATCH /client/{id}`
|
||||
Updates client information.
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"name": "string (max 256 chars)"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /client/{id}/status`
|
||||
Returns client synchronization status.
|
||||
|
||||
**Response Example**:
|
||||
```json
|
||||
{
|
||||
"status": "IN_SYNC" | "NOT_SYNCED" | "NOT_SYNCING"
|
||||
}
|
||||
```
|
||||
|
||||
### Document Operations
|
||||
|
||||
#### `POST /client/{id}/document`
|
||||
Uploads a document for a specific client.
|
||||
|
||||
**Content Type**: `multipart/form-data`
|
||||
**Max File Size**: 100GB
|
||||
**Supported Types**: PDF (extensible)
|
||||
|
||||
**Form Parameters**:
|
||||
- `file`: Document file (required)
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"documentId": "uuid",
|
||||
"clientId": "string",
|
||||
"filename": "string",
|
||||
"uploadedAt": "2024-01-01T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /client/{id}/document`
|
||||
Lists documents for a client with optional filtering.
|
||||
|
||||
**Query Parameters**:
|
||||
- `limit`: Maximum results (integer)
|
||||
- `offset`: Pagination offset (integer)
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"documents": [
|
||||
{
|
||||
"documentId": "uuid",
|
||||
"clientId": "string",
|
||||
"filename": "string",
|
||||
"status": "string",
|
||||
"uploadedAt": "2024-01-01T00:00:00Z"
|
||||
}
|
||||
],
|
||||
"total": "integer"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /document/{id}`
|
||||
Retrieves document metadata by ID.
|
||||
|
||||
**Path Parameters**:
|
||||
- `id`: Document UUID
|
||||
|
||||
### Query Management
|
||||
|
||||
#### `GET /query`
|
||||
Lists all available queries in the system.
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"queries": [
|
||||
{
|
||||
"queryId": "uuid",
|
||||
"queryType": "JSON_EXTRACTOR" | "CONTEXT_FULL",
|
||||
"name": "string",
|
||||
"description": "string",
|
||||
"version": "integer"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### `POST /query`
|
||||
Creates a new query definition.
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"queryType": "JSON_EXTRACTOR" | "CONTEXT_FULL",
|
||||
"name": "string (max 256 chars)",
|
||||
"description": "string (max 512 chars)",
|
||||
"config": "object (JSON configuration)"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /query/{id}`
|
||||
Retrieves query details including configuration.
|
||||
|
||||
#### `PATCH /query/{id}`
|
||||
Updates query definition and increments version.
|
||||
|
||||
#### `POST /query/{id}/test`
|
||||
Tests query execution against sample data.
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"testData": "string",
|
||||
"documentId": "uuid (optional)"
|
||||
}
|
||||
```
|
||||
|
||||
### Collector Configuration
|
||||
|
||||
#### `GET /client/{id}/collector`
|
||||
Retrieves collector configuration for a client.
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"clientId": "string",
|
||||
"queries": [
|
||||
{
|
||||
"name": "string",
|
||||
"queryId": "uuid"
|
||||
}
|
||||
],
|
||||
"minCleanVersion": "integer",
|
||||
"minTextVersion": "integer"
|
||||
}
|
||||
```
|
||||
|
||||
#### `PATCH /client/{id}/collector`
|
||||
Updates collector configuration.
|
||||
|
||||
### Export Operations
|
||||
|
||||
#### `POST /client/{id}/export`
|
||||
Triggers data export for a client.
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"format": "CSV" | "JSON" | "EXCEL",
|
||||
"filters": {
|
||||
"queryIds": ["uuid"],
|
||||
"dateRange": {
|
||||
"start": "2024-01-01T00:00:00Z",
|
||||
"end": "2024-12-31T23:59:59Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /export/{id}`
|
||||
Checks export status and retrieves download URL.
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"exportId": "uuid",
|
||||
"status": "PENDING" | "PROCESSING" | "COMPLETED" | "FAILED",
|
||||
"downloadUrl": "string (when completed)",
|
||||
"createdAt": "2024-01-01T00:00:00Z",
|
||||
"completedAt": "2024-01-01T00:05:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Request/Response Schemas
|
||||
|
||||
### Standard Response Headers
|
||||
All API responses include:
|
||||
- `RateLimit`: Current rate limit status (format: `100;window=60`)
|
||||
- `Content-Type`: `application/json` (except file downloads)
|
||||
|
||||
### Error Response Format
|
||||
```json
|
||||
{
|
||||
"message": "Error description (max 256 chars)"
|
||||
}
|
||||
```
|
||||
|
||||
### Common HTTP Status Codes
|
||||
- `200`: Success
|
||||
- `201`: Created
|
||||
- `400`: Bad Request (validation errors)
|
||||
- `401`: Unauthorized (authentication required)
|
||||
- `403`: Forbidden (insufficient permissions)
|
||||
- `404`: Not Found
|
||||
- `429`: Too Many Requests (rate limited)
|
||||
- `500`: Internal Server Error
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
### Current Limits
|
||||
- **Rate**: 100 requests per 60-second window
|
||||
- **Scope**: Per authenticated user
|
||||
- **Headers**: `RateLimit` header shows current status
|
||||
- **Exceeded**: Returns `429` with `Retry-After` header
|
||||
|
||||
## API Versioning
|
||||
|
||||
### Current Strategy
|
||||
- **URL-based versioning**: `/v1` prefix
|
||||
- **Backward Compatibility**: Maintained within major versions
|
||||
- **OpenAPI Version**: 0.0.1 (development phase)
|
||||
|
||||
### Future Considerations
|
||||
- Semantic versioning for production releases
|
||||
- Deprecation notices for breaking changes
|
||||
- Version sunset policies
|
||||
|
||||
## Code Generation
|
||||
|
||||
### OpenAPI Integration
|
||||
The API is generated from the OpenAPI specification using:
|
||||
```bash
|
||||
oapi-codegen -generate "types,server" -o api.gen.go serviceAPIs/queryAPI.yaml
|
||||
```
|
||||
|
||||
### Generated Artifacts
|
||||
- Type-safe Go structs for requests/responses
|
||||
- Echo framework route handlers
|
||||
- OpenAPI validation middleware
|
||||
- Swagger documentation
|
||||
|
||||
### Validation
|
||||
- Automatic request validation against OpenAPI schema
|
||||
- Response validation in development mode
|
||||
- Parameter binding with type safety
|
||||
@@ -0,0 +1,317 @@
|
||||
# Data Architecture
|
||||
|
||||
## Database Schema Overview
|
||||
|
||||
The system uses PostgreSQL 17.2 with a sophisticated schema designed for multi-tenant document processing, query orchestration, and comprehensive versioning. The schema is managed through database migrations and uses SQLC for type-safe Go integration.
|
||||
|
||||
## Core Entities
|
||||
|
||||
### Clients (`clients`)
|
||||
Central entity representing organizations or tenants in the system.
|
||||
|
||||
```sql
|
||||
CREATE TABLE clients (
|
||||
clientId varchar(256) PRIMARY KEY,
|
||||
name varchar(256) NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
**Key Characteristics**:
|
||||
- String-based client identifiers for human readability
|
||||
- Multi-tenant isolation through client-scoped data
|
||||
- Central reference point for all client-owned entities
|
||||
|
||||
### Documents (`documents`)
|
||||
Represents uploaded files requiring processing.
|
||||
|
||||
```sql
|
||||
CREATE TABLE documents (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||||
clientId varchar(256) NOT NULL REFERENCES clients(clientId),
|
||||
hash varchar(256) NOT NULL,
|
||||
UNIQUE(clientId, hash)
|
||||
);
|
||||
```
|
||||
|
||||
**Key Features**:
|
||||
- UUID v7 primary keys for time-ordered insertion
|
||||
- Client isolation with foreign key constraints
|
||||
- Hash-based deduplication within client scope
|
||||
- Supports document processing pipeline
|
||||
|
||||
### Queries (`queries`)
|
||||
Defines data extraction logic with type-specific implementations.
|
||||
|
||||
```sql
|
||||
CREATE TABLE queries (
|
||||
queryId uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||||
queryType querytype NOT NULL
|
||||
);
|
||||
|
||||
CREATE TYPE querytype AS ENUM ('context_full', 'json_extractor');
|
||||
```
|
||||
|
||||
**Query Types**:
|
||||
- **`context_full`**: Returns complete document context
|
||||
- **`json_extractor`**: Extracts specific JSON paths from structured data
|
||||
|
||||
### Results (`results`)
|
||||
Stores extracted data from document processing.
|
||||
|
||||
```sql
|
||||
CREATE TABLE results (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||||
textEntryId uuid NOT NULL REFERENCES documentTextExtractions(id),
|
||||
queryId uuid NOT NULL REFERENCES queries(queryId),
|
||||
value text NOT NULL,
|
||||
queryVersion int4 NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
## Versioning System
|
||||
|
||||
The schema implements a comprehensive versioning system enabling temporal queries and configuration evolution.
|
||||
|
||||
### Version Management Pattern
|
||||
```sql
|
||||
-- Generic versioning pattern
|
||||
CREATE TABLE entity_versions (
|
||||
entityId uuid NOT NULL,
|
||||
versionId int4 NOT NULL,
|
||||
addedVersion int4 NOT NULL,
|
||||
removedVersion int4,
|
||||
-- entity-specific fields
|
||||
PRIMARY KEY (entityId, versionId)
|
||||
);
|
||||
|
||||
CREATE TABLE entity_active_versions (
|
||||
entityId uuid PRIMARY KEY,
|
||||
versionId int4 NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### Temporal Validity Functions
|
||||
```sql
|
||||
-- Check if version is active at specific point
|
||||
CREATE OR REPLACE FUNCTION isInVersion(addedVersion int4, removedVersion int4, targetVersion int4)
|
||||
RETURNS boolean AS $$
|
||||
BEGIN
|
||||
RETURN addedVersion <= targetVersion AND (removedVersion IS NULL OR removedVersion > targetVersion);
|
||||
END;
|
||||
$$ LANGUAGE plpgsql IMMUTABLE;
|
||||
```
|
||||
|
||||
### Version-Enabled Entities
|
||||
- **Query Versions**: `queryVersions`, `queryActiveVersions`
|
||||
- **Collector Versions**: `collectorVersions`, `collectorActiveVersions`
|
||||
- **Processing Versions**: Track minimum required versions for document stages
|
||||
|
||||
## Document Processing Pipeline
|
||||
|
||||
### Document Lifecycle Tables
|
||||
|
||||
#### Document Uploads (`documentUploads`)
|
||||
Initial upload tracking for audit and debugging.
|
||||
|
||||
#### Document Entries (`documentEntries`)
|
||||
Core document references after deduplication and validation.
|
||||
|
||||
#### Document Cleans (`documentCleans`)
|
||||
PDF validation and format processing records.
|
||||
|
||||
#### Document Text Extractions (`documentTextExtractions`)
|
||||
OCR and text extraction results from AWS Textract.
|
||||
|
||||
### Processing Dependencies
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNo9jz0KwzAMRq9iNDcX6NClZOsU2inNYGy1Mfgn2AokhNy9tkytRTx9TyAdoIJGuAr4RrnM4jG8vcj1WmyQetRBrQ49VUyT6Lqb6D3FvUWFDKap7nHG1t2i9M1i-ksMLD1xo-YU6DeKUpEJTS5TdgdMq6U0xtonuAhwGJ00Ot9_AM3o-BONH5kNOM8fK3VMWw==)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
Upload[documentUploads] --> Entry[documentEntries]
|
||||
Entry --> Clean[documentCleans]
|
||||
Clean --> Text[documentTextExtractions]
|
||||
Text --> Results[results]
|
||||
```
|
||||
|
||||
## Query Dependency System
|
||||
|
||||
### Dependency Management
|
||||
```sql
|
||||
-- Query-to-query dependencies
|
||||
CREATE TABLE requiredQueries (
|
||||
queryId uuid NOT NULL REFERENCES queries(queryId),
|
||||
requiredQueryId uuid NOT NULL REFERENCES queries(queryId),
|
||||
versionId int4 NOT NULL,
|
||||
addedVersion int4 NOT NULL,
|
||||
removedVersion int4,
|
||||
CHECK (queryId != requiredQueryId)
|
||||
);
|
||||
|
||||
-- Result-to-result dependencies
|
||||
CREATE TABLE resultDependencies (
|
||||
resultId uuid NOT NULL REFERENCES results(id),
|
||||
requiredResultId uuid NOT NULL REFERENCES results(id),
|
||||
CHECK (resultId != requiredResultId)
|
||||
);
|
||||
```
|
||||
|
||||
### Dependency Resolution Views
|
||||
```sql
|
||||
-- Recursive view for transitive dependencies
|
||||
CREATE VIEW queryActiveDependencies AS
|
||||
WITH RECURSIVE deps AS (
|
||||
-- Direct dependencies
|
||||
SELECT DISTINCT queryId, requiredQueryId, 1 as depth
|
||||
FROM requiredQueries rq
|
||||
JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
|
||||
WHERE rq.removedVersion IS NULL
|
||||
|
||||
UNION
|
||||
|
||||
-- Transitive dependencies
|
||||
SELECT d.queryId, rq.requiredQueryId, d.depth + 1
|
||||
FROM deps d
|
||||
JOIN requiredQueries rq ON d.requiredQueryId = rq.queryId
|
||||
JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
|
||||
WHERE rq.removedVersion IS NULL AND d.depth < 10 -- Cycle prevention
|
||||
)
|
||||
SELECT * FROM deps;
|
||||
```
|
||||
|
||||
## Collector Configuration System
|
||||
|
||||
### Client-Query Mapping
|
||||
```sql
|
||||
-- Maps named queries to specific query implementations for clients
|
||||
CREATE TABLE collectorQueries (
|
||||
clientId varchar(256) NOT NULL REFERENCES clients(clientId),
|
||||
name varchar(256) NOT NULL,
|
||||
queryId uuid NOT NULL REFERENCES queries(queryId),
|
||||
versionId int4 NOT NULL,
|
||||
addedVersion int4 NOT NULL,
|
||||
removedVersion int4
|
||||
);
|
||||
```
|
||||
|
||||
### Version Thresholds
|
||||
```sql
|
||||
-- Minimum processing versions required
|
||||
CREATE TABLE collectorMinCleanVersions (
|
||||
clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId),
|
||||
version int4 NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE collectorMinTextVersions (
|
||||
clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId),
|
||||
version int4 NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### Complex Dependency Views
|
||||
```sql
|
||||
-- Builds complete query dependency tree per client
|
||||
CREATE VIEW collectorQueryDependencyTree AS
|
||||
SELECT DISTINCT
|
||||
cq.clientId,
|
||||
cq.name as queryName,
|
||||
qad.queryId,
|
||||
qad.requiredQueryId,
|
||||
qad.depth
|
||||
FROM collectorQueries cq
|
||||
JOIN queryActiveDependencies qad ON cq.queryId = qad.queryId
|
||||
WHERE cq.removedVersion IS NULL;
|
||||
```
|
||||
|
||||
## Advanced Database Features
|
||||
|
||||
### Business Logic Functions
|
||||
|
||||
#### Document Processing Functions
|
||||
```sql
|
||||
-- List documents ready for processing
|
||||
CREATE OR REPLACE FUNCTION listDocumentIDs(clientId varchar(256), limit_val int4, offset_val int4)
|
||||
RETURNS TABLE(documentId uuid) AS $$
|
||||
BEGIN
|
||||
RETURN QUERY
|
||||
SELECT de.documentId
|
||||
FROM documentEntries de
|
||||
WHERE de.clientId = $1
|
||||
ORDER BY de.documentId
|
||||
LIMIT limit_val OFFSET offset_val;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
```
|
||||
|
||||
#### Result Validation Functions
|
||||
```sql
|
||||
-- Complex validation for result dependencies
|
||||
CREATE OR REPLACE FUNCTION listValidDocumentResults(
|
||||
clientId varchar(256),
|
||||
documentId uuid,
|
||||
queryId uuid
|
||||
)
|
||||
RETURNS TABLE(resultId uuid, value text) AS $$
|
||||
-- Complex logic for dependency resolution
|
||||
$$ LANGUAGE plpgsql;
|
||||
```
|
||||
|
||||
### Materialized Views and Performance
|
||||
```sql
|
||||
-- Current clean entries above minimum thresholds
|
||||
CREATE VIEW currentCleanEntries AS
|
||||
SELECT DISTINCT ON (dc.documentId)
|
||||
dc.*
|
||||
FROM documentCleans dc
|
||||
JOIN collectorMinCleanVersions cmcv ON dc.clientId = cmcv.clientId
|
||||
WHERE dc.version >= cmcv.version
|
||||
ORDER BY dc.documentId, dc.version DESC;
|
||||
```
|
||||
|
||||
## Data Integrity and Constraints
|
||||
|
||||
### Referential Integrity
|
||||
- All foreign key relationships enforced at database level
|
||||
- Cascade delete policies for cleanup operations
|
||||
- Unique constraints prevent duplicate business entities
|
||||
|
||||
### Data Validation
|
||||
- Enum types for controlled vocabularies (`querytype`, `mimetype`)
|
||||
- Check constraints for business rules (no self-dependencies)
|
||||
- NOT NULL constraints for required fields
|
||||
|
||||
### Audit and Traceability
|
||||
- UUID v7 generation provides insertion ordering
|
||||
- Version tracking enables complete audit trails
|
||||
- Temporal validity supports point-in-time queries
|
||||
|
||||
## Scalability Considerations
|
||||
|
||||
### Indexing Strategy
|
||||
- Primary keys automatically indexed (UUID v7)
|
||||
- Foreign keys automatically indexed
|
||||
- Composite indexes on frequent query patterns
|
||||
- Unique constraints create secondary indexes
|
||||
|
||||
### Partitioning Opportunities
|
||||
- Time-based partitioning using UUID v7 timestamp component
|
||||
- Client-based partitioning for multi-tenant isolation
|
||||
- Archive strategies for old versions
|
||||
|
||||
### Performance Optimizations
|
||||
- Window functions for latest version queries
|
||||
- Recursive CTEs with depth limits for dependency resolution
|
||||
- Pagination support in all list functions
|
||||
- Efficient temporal queries using `isInVersion()` function
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Database Evolution
|
||||
- Sequential numbered migrations for version control
|
||||
- Separate UP/DOWN migrations for rollback capability
|
||||
- Extension management for UUID generation and other features
|
||||
- View-based abstractions for query flexibility
|
||||
|
||||
### Version Compatibility
|
||||
- Backward-compatible schema changes
|
||||
- Graceful handling of version mismatches
|
||||
- Data migration scripts for major version upgrades
|
||||
@@ -0,0 +1,383 @@
|
||||
# Getting Started Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Required Tools
|
||||
- **Devbox**: Provides reproducible development environment
|
||||
- Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash`
|
||||
- NixOS: `devbox` (already available)
|
||||
- **Docker**: Container runtime for services and testing
|
||||
- Install from: https://docs.docker.com/engine/install/
|
||||
- **Git**: Version control (usually pre-installed)
|
||||
|
||||
### System Requirements
|
||||
- **OS**: Linux, macOS, or Windows with WSL2
|
||||
- **RAM**: 8GB minimum, 16GB recommended
|
||||
- **Disk**: 10GB free space for development environment
|
||||
- **Network**: Internet access for downloading dependencies
|
||||
|
||||
### Docker Group Setup (Linux)
|
||||
Ensure your user can run Docker commands without sudo:
|
||||
```bash
|
||||
sudo groupadd docker
|
||||
sudo usermod -aG docker $USER
|
||||
# Log out and back in for changes to take effect
|
||||
```
|
||||
|
||||
## Local Development Setup
|
||||
|
||||
### 1. Environment Initialization
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone <repository-url>
|
||||
cd query-orchestration.2
|
||||
|
||||
# Create environment file for custom variables
|
||||
touch .env
|
||||
|
||||
# Enter development environment (installs all tools automatically)
|
||||
devbox shell
|
||||
```
|
||||
|
||||
### 2. Verify Installation
|
||||
```bash
|
||||
# Check Go version
|
||||
go version # Should show 1.24.0
|
||||
|
||||
# Check Docker
|
||||
docker --version
|
||||
|
||||
# Check Task runner
|
||||
task --version # Should show 3.39.2+
|
||||
|
||||
# List available tasks
|
||||
task --list
|
||||
```
|
||||
|
||||
### 3. Full System Validation
|
||||
Run the complete test suite to ensure everything works:
|
||||
```bash
|
||||
task fullsuite
|
||||
```
|
||||
|
||||
This command will:
|
||||
1. Generate latest API specifications
|
||||
2. Run linting commands
|
||||
3. Execute unit tests
|
||||
4. Run end-to-end integration tests
|
||||
5. Validate coverage thresholds (80% total, 60% function)
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Common Development Tasks
|
||||
|
||||
#### Running Services Locally
|
||||
```bash
|
||||
# Start all services with local infrastructure
|
||||
task dev:up
|
||||
|
||||
# Start specific service
|
||||
task dev:api # Just the API service
|
||||
task dev:runners # Just the queue runners
|
||||
```
|
||||
|
||||
#### API Development
|
||||
```bash
|
||||
# Generate API code from OpenAPI spec
|
||||
task openapi:generate
|
||||
|
||||
# Start API server with hot reload
|
||||
task dev:api
|
||||
|
||||
# View API documentation
|
||||
open http://localhost:8080/swagger/index.html
|
||||
```
|
||||
|
||||
#### Database Operations
|
||||
```bash
|
||||
# Run database migrations
|
||||
task db:migrate
|
||||
|
||||
# Reset database (development only)
|
||||
task db:reset
|
||||
|
||||
# Generate database models from SQL
|
||||
task db:generate
|
||||
```
|
||||
|
||||
#### Testing
|
||||
```bash
|
||||
# Run unit tests
|
||||
task test:unit
|
||||
|
||||
# Run integration tests
|
||||
task test:integration
|
||||
|
||||
# Run tests with coverage
|
||||
task test:coverage
|
||||
|
||||
# Run specific test package
|
||||
go test ./internal/client/...
|
||||
```
|
||||
|
||||
#### Code Quality
|
||||
```bash
|
||||
# Run linters
|
||||
task lint
|
||||
|
||||
# Format code
|
||||
task format
|
||||
|
||||
# Generate mocks for testing
|
||||
task generate:mocks
|
||||
```
|
||||
|
||||
### Development Environment Configuration
|
||||
|
||||
#### Environment Variables Hierarchy
|
||||
1. **devbox.json "env"** section (highest priority)
|
||||
2. **devbox.json "init_hook"** exports
|
||||
3. **.env file** variables (lowest priority)
|
||||
|
||||
#### Key Development Variables
|
||||
Create a `.env` file with:
|
||||
```bash
|
||||
# Database (optional - defaults provided)
|
||||
PGUSER=postgres
|
||||
PGPASSWORD=password
|
||||
PGDATABASE=queryorchestration
|
||||
|
||||
# AWS (uses LocalStack in development)
|
||||
AWS_REGION=us-east-1
|
||||
AWS_ACCESS_KEY_ID=test
|
||||
AWS_SECRET_ACCESS_KEY=test
|
||||
|
||||
# Authentication (disable for local development)
|
||||
DISABLE_AUTH=true
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=DEBUG
|
||||
```
|
||||
|
||||
#### Service Ports (Local Development)
|
||||
- **queryAPI**: 8080 (main API)
|
||||
- **storeEventRunner**: 8081
|
||||
- **docInitRunner**: 8082
|
||||
- **docSyncRunner**: 8083
|
||||
- **docCleanRunner**: 8084
|
||||
- **docTextRunner**: 8085
|
||||
- **querySyncRunner**: 8087
|
||||
- **queryRunner**: 8088
|
||||
- **clientSyncRunner**: 8089
|
||||
- **queryVersionSyncRunner**: 8090
|
||||
- **Prometheus**: 9091
|
||||
- **PostgreSQL**: 5432
|
||||
- **LocalStack**: 4566
|
||||
|
||||
## Common Development Workflows
|
||||
|
||||
### 1. API Endpoint Development
|
||||
```bash
|
||||
# 1. Modify OpenAPI specification
|
||||
vim serviceAPIs/queryAPI.yaml
|
||||
|
||||
# 2. Regenerate API code
|
||||
task openapi:generate
|
||||
|
||||
# 3. Implement controller logic
|
||||
vim api/queryAPI/controllers.go
|
||||
|
||||
# 4. Run API server
|
||||
task dev:api
|
||||
|
||||
# 5. Test endpoint
|
||||
curl -X GET http://localhost:8080/health
|
||||
```
|
||||
|
||||
### 2. Database Schema Changes
|
||||
```bash
|
||||
# 1. Create new migration files
|
||||
vim internal/database/migrations/000000000000X_feature.up.sql
|
||||
vim internal/database/migrations/000000000000X_feature.down.sql
|
||||
|
||||
# 2. Add queries to SQL files
|
||||
vim internal/database/queries/feature.sql
|
||||
|
||||
# 3. Regenerate database models
|
||||
task db:generate
|
||||
|
||||
# 4. Run migrations
|
||||
task db:migrate
|
||||
|
||||
# 5. Update Go service code
|
||||
vim internal/feature/service.go
|
||||
```
|
||||
|
||||
### 3. New Service Development
|
||||
```bash
|
||||
# 1. Implement controller interface
|
||||
vim api/newRunner/runner.go
|
||||
|
||||
# 2. Create main command
|
||||
vim cmd/newRunner/main.go
|
||||
|
||||
# 3. Add service configuration
|
||||
vim internal/serviceconfig/newrunner/config.go
|
||||
|
||||
# 4. Update docker-compose
|
||||
vim deployments/compose.local.yaml
|
||||
|
||||
# 5. Run tests
|
||||
task test:integration
|
||||
```
|
||||
|
||||
### 4. Adding New Queue
|
||||
```bash
|
||||
# 1. Add queue configuration
|
||||
vim internal/serviceconfig/queue/newqueue/config.go
|
||||
|
||||
# 2. Update producer services
|
||||
vim internal/service/producer.go
|
||||
|
||||
# 3. Update consumer services
|
||||
vim api/consumerRunner/runner.go
|
||||
|
||||
# 4. Update environment variables
|
||||
vim deployments/compose.local.yaml
|
||||
```
|
||||
|
||||
## Testing Guidelines
|
||||
|
||||
### Unit Testing
|
||||
- **Location**: `*_test.go` files alongside source code
|
||||
- **Pattern**: Table-driven tests preferred
|
||||
- **Mocking**: Use generated mocks for external dependencies
|
||||
- **Coverage**: Minimum 60% function coverage required
|
||||
|
||||
Example unit test:
|
||||
```go
|
||||
func TestService_CreateClient(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input CreateClientRequest
|
||||
expected Client
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "valid client creation",
|
||||
input: CreateClientRequest{
|
||||
ClientID: "test-client",
|
||||
Name: "Test Client",
|
||||
},
|
||||
expected: Client{
|
||||
ClientID: "test-client",
|
||||
Name: "Test Client",
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
// Test implementation
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration Testing
|
||||
- **Location**: `test/` directory and `*_test.go` files
|
||||
- **Infrastructure**: Uses testcontainers for real dependencies
|
||||
- **Isolation**: Each test gets clean environment
|
||||
- **Coverage**: Minimum 80% total coverage required
|
||||
|
||||
### End-to-End Testing
|
||||
- **Environment**: Full service deployment via Docker Compose
|
||||
- **Scope**: Complete workflows from API to database
|
||||
- **Authentication**: Uses test Cognito setup or disabled auth
|
||||
|
||||
## Debugging and Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Docker Permission Errors
|
||||
```bash
|
||||
# Fix Docker permissions (Linux)
|
||||
sudo groupadd docker
|
||||
sudo usermod -aG docker $USER
|
||||
# Log out and log back in
|
||||
```
|
||||
|
||||
#### Database Connection Issues
|
||||
```bash
|
||||
# Check PostgreSQL status
|
||||
task db:status
|
||||
|
||||
# Reset database
|
||||
task db:reset
|
||||
|
||||
# Check connection manually
|
||||
psql -h localhost -p 5432 -U postgres -d queryorchestration
|
||||
```
|
||||
|
||||
#### Service Startup Issues
|
||||
```bash
|
||||
# Check service logs
|
||||
docker-compose -f deployments/compose.local.yaml logs [service-name]
|
||||
|
||||
# Check environment variables
|
||||
task env:check
|
||||
|
||||
# Verify ports aren't in use
|
||||
lsof -i :8080
|
||||
```
|
||||
|
||||
#### Code Generation Issues
|
||||
```bash
|
||||
# Clean generated files
|
||||
task clean
|
||||
|
||||
# Regenerate everything
|
||||
task generate:all
|
||||
|
||||
# Check for syntax errors in source files
|
||||
task lint
|
||||
```
|
||||
|
||||
### Development Tools
|
||||
|
||||
#### Debugging
|
||||
- **Delve**: Go debugger integrated with devbox
|
||||
- **IDE Integration**: VS Code and GoLand support provided
|
||||
- **Logging**: Structured logging with slog at DEBUG level
|
||||
|
||||
#### Monitoring
|
||||
- **Prometheus**: Metrics at http://localhost:9091
|
||||
- **Swagger UI**: API docs at http://localhost:8080/swagger/
|
||||
- **Health Checks**: http://localhost:8080/health
|
||||
|
||||
#### Database Tools
|
||||
- **psql**: Command-line PostgreSQL client
|
||||
- **Database browser**: Connect to localhost:5432
|
||||
- **Migration status**: `task db:status`
|
||||
|
||||
### Performance Profiling
|
||||
```bash
|
||||
# CPU profiling
|
||||
go test -cpuprofile=cpu.prof -bench=.
|
||||
|
||||
# Memory profiling
|
||||
go test -memprofile=mem.prof -bench=.
|
||||
|
||||
# Race detection
|
||||
go test -race ./...
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
After completing the setup:
|
||||
1. **Explore the API**: Visit http://localhost:8080/swagger/
|
||||
2. **Review Architecture**: Read the service architecture documentation
|
||||
3. **Check Data Model**: Examine database schema documentation
|
||||
4. **Run Examples**: Try the metrics example service
|
||||
5. **Write Tests**: Create tests for new features
|
||||
@@ -0,0 +1,389 @@
|
||||
# Code Organization
|
||||
|
||||
## Project Structure
|
||||
|
||||
The codebase follows the [Standard Go Project Layout](https://github.com/golang-standards/project-layout) with additional conventions specific to the domain requirements.
|
||||
|
||||
### Directory Overview
|
||||
|
||||
```
|
||||
query-orchestration.2/
|
||||
├── cmd/ # Main applications and entry points
|
||||
├── internal/ # Private application code
|
||||
├── api/ # Generated API server implementations
|
||||
├── pkg/ # Public library code
|
||||
├── deployments/ # Deployment configurations
|
||||
├── docs/ # Documentation
|
||||
├── scripts/ # Build and automation scripts
|
||||
├── test/ # Integration and e2e tests
|
||||
├── mocks/ # Generated mock implementations
|
||||
├── assets/ # Test data and sample files
|
||||
└── vendor/ # Vendored dependencies
|
||||
```
|
||||
|
||||
## Package Architecture
|
||||
|
||||
### Command Layer (`cmd/`)
|
||||
|
||||
Entry points for all deployable services, following the pattern:
|
||||
```
|
||||
cmd/
|
||||
├── [serviceName]/
|
||||
│ └── main.go # Service bootstrap and configuration
|
||||
```
|
||||
|
||||
**Naming Conventions**:
|
||||
- **APIs**: `[functionality]API` (e.g., `queryAPI`)
|
||||
- **Runners**: `[functionality]Runner` (e.g., `docInitRunner`)
|
||||
- **Utilities**: Descriptive names (e.g., `healthcheck`)
|
||||
|
||||
### Internal Packages (`internal/`)
|
||||
|
||||
Core business logic organized by domain boundaries:
|
||||
|
||||
#### Domain Services
|
||||
```
|
||||
internal/
|
||||
├── client/ # Client management domain
|
||||
├── document/ # Document processing domain
|
||||
├── query/ # Query definition and execution
|
||||
├── collector/ # Client-specific configurations
|
||||
└── export/ # Data export operations
|
||||
```
|
||||
|
||||
#### Infrastructure Services
|
||||
```
|
||||
internal/
|
||||
├── database/ # Data persistence layer
|
||||
├── server/ # HTTP and queue server infrastructure
|
||||
├── serviceconfig/ # Configuration management
|
||||
├── cognitoauth/ # Authentication middleware
|
||||
├── test/ # Testing utilities
|
||||
└── validation/ # Input validation
|
||||
```
|
||||
|
||||
### API Layer (`api/`)
|
||||
|
||||
Generated server implementations from OpenAPI specifications:
|
||||
```
|
||||
api/
|
||||
├── queryAPI/ # REST API implementation
|
||||
│ ├── api.gen.go # Generated types and interfaces
|
||||
│ ├── controllers.go # Business logic implementation
|
||||
│ ├── authHandlers.go # Authentication endpoints
|
||||
│ └── *.go # Domain-specific handlers
|
||||
└── [runnerName]/ # Queue-based service implementations
|
||||
└── runner.go # Queue message processing
|
||||
```
|
||||
|
||||
## Design Patterns and Principles
|
||||
|
||||
### Service Layer Pattern
|
||||
|
||||
Each domain follows a consistent service layer structure:
|
||||
|
||||
```go
|
||||
// Service struct with injected dependencies
|
||||
type Service struct {
|
||||
cfg ConfigProvider // Configuration interface
|
||||
}
|
||||
|
||||
// Constructor with interface-based dependency injection
|
||||
func New(cfg ConfigProvider) *Service {
|
||||
return &Service{cfg: cfg}
|
||||
}
|
||||
|
||||
// Business operations
|
||||
func (s *Service) CreateEntity(ctx context.Context, req CreateRequest) (*Entity, error) {
|
||||
// Business logic implementation
|
||||
}
|
||||
```
|
||||
|
||||
### Repository Pattern
|
||||
|
||||
Database access is abstracted through repositories:
|
||||
|
||||
```go
|
||||
// Generated by SQLC from SQL queries
|
||||
type Repository interface {
|
||||
CreateClient(ctx context.Context, arg CreateClientParams) (Client, error)
|
||||
GetClient(ctx context.Context, clientID string) (Client, error)
|
||||
// ... other operations
|
||||
}
|
||||
|
||||
// Service uses repository interface
|
||||
type Service struct {
|
||||
repo Repository
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Provider Pattern
|
||||
|
||||
Services receive configuration through composed interfaces:
|
||||
|
||||
```go
|
||||
// Domain-specific configuration interface
|
||||
type ConfigProvider interface {
|
||||
serviceconfig.ConfigProvider // Base configuration
|
||||
database.ConfigProvider // Database access
|
||||
objectstore.ConfigProvider // S3 access
|
||||
// ... other provider interfaces
|
||||
}
|
||||
|
||||
// Concrete configuration embeds all providers
|
||||
type Config struct {
|
||||
serviceconfig.BaseConfig
|
||||
database.Config
|
||||
objectstore.Config
|
||||
}
|
||||
```
|
||||
|
||||
### Dependency Injection
|
||||
|
||||
Services are composed through constructor injection:
|
||||
|
||||
```go
|
||||
// In main.go
|
||||
cfg := &Config{
|
||||
BaseConfig: baseConfig,
|
||||
DatabaseConfig: dbConfig,
|
||||
// ... other configs
|
||||
}
|
||||
|
||||
// Create service with injected dependencies
|
||||
svc := service.New(cfg)
|
||||
|
||||
// Create controller with service dependency
|
||||
controller := api.NewController(svc)
|
||||
```
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
### Package Names
|
||||
- **Domain packages**: Singular nouns (`client`, `document`, `query`)
|
||||
- **Utility packages**: Descriptive (`serviceconfig`, `cognitoauth`)
|
||||
- **Generated packages**: Match source (`queryAPI` from `queryAPI.yaml`)
|
||||
|
||||
### File Names
|
||||
- **Service files**: `service.go` (main business logic)
|
||||
- **Test files**: `*_test.go` (alongside source)
|
||||
- **Private tests**: `*private_test.go` (package-private testing)
|
||||
- **Interface files**: Match domain (`client.go`, `document.go`)
|
||||
|
||||
### Function and Method Names
|
||||
- **Public functions**: PascalCase (`CreateClient`, `GetDocument`)
|
||||
- **Private functions**: camelCase (`validateInput`, `processResult`)
|
||||
- **Constructors**: `New` or `New[Type]` (`New`, `NewService`)
|
||||
- **Getters**: No `Get` prefix for simple accessors (`ID()`, not `GetID()`)
|
||||
|
||||
### Variable Names
|
||||
- **Short scope**: Single letters (`i`, `j`, `c` for client)
|
||||
- **Medium scope**: Abbreviations (`ctx`, `req`, `resp`)
|
||||
- **Long scope**: Descriptive names (`clientRepository`, `configProvider`)
|
||||
- **Constants**: PascalCase (`MaxRetryAttempts`, `DefaultTimeout`)
|
||||
|
||||
### Type Names
|
||||
- **Structs**: PascalCase nouns (`Client`, `DocumentProcessor`)
|
||||
- **Interfaces**: PascalCase with optional "er" suffix (`ConfigProvider`, `Processor`)
|
||||
- **Enums**: PascalCase with type suffix (`QueryType`, `ProcessingStatus`)
|
||||
|
||||
## Code Style Guidelines
|
||||
|
||||
### Import Organization
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
// Standard library
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
// Third-party packages
|
||||
"github.com/labstack/echo/v4"
|
||||
"github.com/google/uuid"
|
||||
|
||||
// Local packages
|
||||
"queryorchestration/internal/client"
|
||||
"queryorchestration/internal/database"
|
||||
)
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
```go
|
||||
// Wrap errors with context
|
||||
func (s *Service) ProcessDocument(ctx context.Context, id uuid.UUID) error {
|
||||
doc, err := s.repo.GetDocument(ctx, id)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get document %s: %w", id, err)
|
||||
}
|
||||
|
||||
if err := s.processContent(doc); err != nil {
|
||||
return fmt.Errorf("failed to process document content: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
### Interface Design
|
||||
```go
|
||||
// Prefer small, focused interfaces
|
||||
type ClientReader interface {
|
||||
GetClient(ctx context.Context, id string) (*Client, error)
|
||||
}
|
||||
|
||||
type ClientWriter interface {
|
||||
CreateClient(ctx context.Context, client *Client) error
|
||||
UpdateClient(ctx context.Context, client *Client) error
|
||||
}
|
||||
|
||||
// Compose when needed
|
||||
type ClientRepository interface {
|
||||
ClientReader
|
||||
ClientWriter
|
||||
}
|
||||
```
|
||||
|
||||
### Struct Design
|
||||
```go
|
||||
// Group related fields
|
||||
type Client struct {
|
||||
// Identity
|
||||
ID string
|
||||
Name string
|
||||
|
||||
// Configuration
|
||||
CanSync bool
|
||||
|
||||
// Metadata
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
// Use embedding for composition
|
||||
type APIClient struct {
|
||||
Client
|
||||
AuthToken string
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Patterns
|
||||
|
||||
### Table-Driven Tests
|
||||
```go
|
||||
func TestService_CreateClient(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input CreateClientRequest
|
||||
want *Client
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "valid client",
|
||||
input: CreateClientRequest{
|
||||
ID: "test-client",
|
||||
Name: "Test Client",
|
||||
},
|
||||
want: &Client{
|
||||
ID: "test-client",
|
||||
Name: "Test Client",
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "empty ID",
|
||||
input: CreateClientRequest{
|
||||
Name: "Test Client",
|
||||
},
|
||||
want: nil,
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
// Test implementation
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Mock Usage
|
||||
```go
|
||||
// Generate mocks for interfaces
|
||||
//go:generate mockery --name=ClientRepository --output=mocks
|
||||
|
||||
func TestService_WithMocks(t *testing.T) {
|
||||
// Create mock
|
||||
mockRepo := &mocks.ClientRepository{}
|
||||
|
||||
// Set expectations
|
||||
mockRepo.On("GetClient", mock.Anything, "test-id").
|
||||
Return(&Client{ID: "test-id"}, nil)
|
||||
|
||||
// Create service with mock
|
||||
svc := &Service{repo: mockRepo}
|
||||
|
||||
// Test and verify
|
||||
result, err := svc.GetClient(context.Background(), "test-id")
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "test-id", result.ID)
|
||||
|
||||
mockRepo.AssertExpectations(t)
|
||||
}
|
||||
```
|
||||
|
||||
### Integration Test Structure
|
||||
```go
|
||||
func TestIntegration_DocumentProcessing(t *testing.T) {
|
||||
// Setup test environment
|
||||
ctx := context.Background()
|
||||
testDB := test.NewDatabase(t)
|
||||
testS3 := test.NewObjectStore(t)
|
||||
|
||||
// Create service with real dependencies
|
||||
cfg := &Config{
|
||||
Database: testDB,
|
||||
ObjectStore: testS3,
|
||||
}
|
||||
svc := New(cfg)
|
||||
|
||||
// Run test scenario
|
||||
client := test.CreateTestClient(t, svc)
|
||||
document := test.UploadTestDocument(t, svc, client.ID)
|
||||
|
||||
// Verify results
|
||||
result := test.ProcessDocument(t, svc, document.ID)
|
||||
assert.NotNil(t, result)
|
||||
|
||||
// Cleanup handled by test utilities
|
||||
}
|
||||
```
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### Configuration Management
|
||||
- **Environment-first**: All configuration from environment variables
|
||||
- **Type safety**: Strong typing with validation
|
||||
- **Composition**: Interface-based configuration providers
|
||||
- **Testing**: Override-friendly for test scenarios
|
||||
|
||||
### Error Handling
|
||||
- **Wrap errors**: Add context at each layer
|
||||
- **Structured logging**: Use slog for consistent error logging
|
||||
- **Graceful degradation**: Continue operation when possible
|
||||
- **Client-friendly**: Translate internal errors to appropriate HTTP status
|
||||
|
||||
### Database Access
|
||||
- **Type safety**: SQLC generates type-safe database code
|
||||
- **Migrations**: Version-controlled schema evolution
|
||||
- **Connection pooling**: Shared connections across services
|
||||
- **Transaction management**: Explicit transaction boundaries
|
||||
|
||||
### API Design
|
||||
- **Contract-first**: OpenAPI specification drives implementation
|
||||
- **Version compatibility**: Backward-compatible changes
|
||||
- **Validation**: Automatic request/response validation
|
||||
- **Documentation**: Auto-generated from specification
|
||||
|
||||
This organization enables maintainable, testable, and scalable code while following Go best practices and domain-driven design principles.
|
||||
@@ -0,0 +1,405 @@
|
||||
# Configuration Management
|
||||
|
||||
## Configuration Architecture
|
||||
|
||||
The system implements a hierarchical, environment-driven configuration system designed for flexibility, type safety, and testability. Configuration is managed through the `internal/serviceconfig` package with domain-specific sub-packages.
|
||||
|
||||
## Configuration Hierarchy
|
||||
|
||||
### Environment Variable Priority (First Wins)
|
||||
1. **devbox.json "env"** section - Highest priority for development
|
||||
2. **devbox.json "init_hook"** exports - Build-time configuration
|
||||
3. **.env file** variables - Local overrides
|
||||
4. **System environment** - Runtime environment variables
|
||||
|
||||
### Configuration Loading Process
|
||||
```go
|
||||
// 1. Load .env file if present (development)
|
||||
godotenv.Load()
|
||||
|
||||
// 2. Parse environment variables into struct
|
||||
env.Parse(&config)
|
||||
|
||||
// 3. Validate required fields
|
||||
config.Validate()
|
||||
|
||||
// 4. Initialize dependent services
|
||||
config.Initialize(ctx)
|
||||
```
|
||||
|
||||
## Base Configuration Structure
|
||||
|
||||
### Core Configuration (`serviceconfig/common.go`)
|
||||
```go
|
||||
type BaseConfig struct {
|
||||
// Application
|
||||
LogLevel string `env:"LOG_LEVEL" envDefault:"INFO"`
|
||||
|
||||
// Server
|
||||
Port int `env:"PORT" envDefault:"8080"`
|
||||
HTTPHost string `env:"HTTP_HOST" envDefault:"0.0.0.0"`
|
||||
|
||||
// Feature Flags
|
||||
DisableAuth bool `env:"DISABLE_AUTH" envDefault:"false"`
|
||||
EnableOTEL bool `env:"ENABLE_OTEL" envDefault:"false"`
|
||||
|
||||
// Observability
|
||||
MetricsPath string `env:"METRICS_PATH" envDefault:"/metrics"`
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Providers Pattern
|
||||
```go
|
||||
// Interface composition for service dependencies
|
||||
type ConfigProvider interface {
|
||||
GetLogLevel() string
|
||||
GetPort() int
|
||||
GetHTTPHost() string
|
||||
IsAuthDisabled() bool
|
||||
}
|
||||
|
||||
// Service-specific configuration embeds base
|
||||
type ServiceConfig struct {
|
||||
BaseConfig
|
||||
DatabaseConfig database.Config
|
||||
AWSConfig aws.Config
|
||||
QueueConfig queue.Config
|
||||
}
|
||||
```
|
||||
|
||||
## Domain-Specific Configuration
|
||||
|
||||
### Database Configuration (`serviceconfig/database/`)
|
||||
```go
|
||||
type Config struct {
|
||||
// Connection
|
||||
Host string `env:"PGHOST" envDefault:"localhost"`
|
||||
Port int `env:"PGPORT" envDefault:"5432"`
|
||||
Database string `env:"PGDATABASE"`
|
||||
Username string `env:"PGUSER"`
|
||||
Password string `env:"PGPASSWORD"`
|
||||
|
||||
// Connection Pool
|
||||
MaxOpenConns int `env:"DB_MAX_OPEN_CONNS" envDefault:"25"`
|
||||
MaxIdleConns int `env:"DB_MAX_IDLE_CONNS" envDefault:"25"`
|
||||
ConnMaxLifetime int `env:"DB_CONN_MAX_LIFETIME" envDefault:"300"`
|
||||
|
||||
// Security
|
||||
SSLMode string `env:"DB_SSL_MODE" envDefault:"require"`
|
||||
NoSSL bool `env:"DB_NOSSL" envDefault:"false"`
|
||||
}
|
||||
|
||||
// Provider interface
|
||||
type ConfigProvider interface {
|
||||
GetDatabaseConfig() Config
|
||||
}
|
||||
|
||||
// Connection pool management
|
||||
func (c Config) CreatePool(ctx context.Context) (*pgxpool.Pool, error) {
|
||||
dsn := c.buildConnectionString()
|
||||
return pgxpool.New(ctx, dsn)
|
||||
}
|
||||
```
|
||||
|
||||
### AWS Configuration (`serviceconfig/aws/`)
|
||||
```go
|
||||
type Config struct {
|
||||
// Credentials
|
||||
Region string `env:"AWS_REGION" envDefault:"us-east-1"`
|
||||
AccessKeyID string `env:"AWS_ACCESS_KEY_ID"`
|
||||
SecretAccessKey string `env:"AWS_SECRET_ACCESS_KEY"`
|
||||
SessionToken string `env:"AWS_SESSION_TOKEN"`
|
||||
Profile string `env:"AWS_PROFILE"`
|
||||
|
||||
// Endpoints (for LocalStack/testing)
|
||||
EndpointURL string `env:"AWS_ENDPOINT_URL"`
|
||||
S3UsePathStyle bool `env:"AWS_S3_USE_PATH_STYLE" envDefault:"false"`
|
||||
|
||||
// Service-specific
|
||||
S3Bucket string `env:"S3_BUCKET"`
|
||||
TextractRoleARN string `env:"TEXTRACT_ROLE_ARN"`
|
||||
}
|
||||
|
||||
// SDK integration
|
||||
func (c Config) LoadAWSConfig(ctx context.Context) (aws.Config, error) {
|
||||
var opts []func(*config.LoadOptions) error
|
||||
|
||||
if c.Region != "" {
|
||||
opts = append(opts, config.WithRegion(c.Region))
|
||||
}
|
||||
|
||||
if c.EndpointURL != "" {
|
||||
opts = append(opts, config.WithEndpointResolverWithOptions(
|
||||
aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
|
||||
return aws.Endpoint{URL: c.EndpointURL}, nil
|
||||
}),
|
||||
))
|
||||
}
|
||||
|
||||
return config.LoadDefaultConfig(ctx, opts...)
|
||||
}
|
||||
```
|
||||
|
||||
### Queue Configuration (`serviceconfig/queue/`)
|
||||
```go
|
||||
// Base queue configuration
|
||||
type Config struct {
|
||||
// Connection
|
||||
QueueURL string `env:"QUEUE_URL"`
|
||||
MaxMessages int32 `env:"QUEUE_MAX_MESSAGES" envDefault:"10"`
|
||||
WaitTimeSeconds int32 `env:"QUEUE_WAIT_TIME" envDefault:"20"`
|
||||
VisibilityTimeout int32 `env:"QUEUE_VISIBILITY_TIMEOUT" envDefault:"30"`
|
||||
|
||||
// Processing
|
||||
WorkerCount int `env:"QUEUE_WORKER_COUNT" envDefault:"5"`
|
||||
MaxRetries int `env:"QUEUE_MAX_RETRIES" envDefault:"3"`
|
||||
RetryDelay int `env:"QUEUE_RETRY_DELAY" envDefault:"5"`
|
||||
}
|
||||
|
||||
// Service-specific queue configs
|
||||
type DocumentInitConfig struct {
|
||||
Config
|
||||
DocumentSyncURL string `env:"DOCUMENT_SYNC_URL"`
|
||||
}
|
||||
|
||||
type QueryConfig struct {
|
||||
Config
|
||||
QuerySyncURL string `env:"QUERY_SYNC_URL"`
|
||||
}
|
||||
```
|
||||
|
||||
### Authentication Configuration (`serviceconfig/auth/`)
|
||||
```go
|
||||
type Config struct {
|
||||
// Cognito Configuration
|
||||
UserPoolID string `env:"COGNITO_USER_POOL_ID"`
|
||||
ClientID string `env:"COGNITO_CLIENT_ID"`
|
||||
ClientSecret string `env:"COGNITO_CLIENT_SECRET" envSecret:"true"`
|
||||
Domain string `env:"COGNITO_DOMAIN"`
|
||||
|
||||
// URLs
|
||||
RedirectURL string `env:"COGNITO_REDIRECT_URL"`
|
||||
LogoutURL string `env:"COGNITO_LOGOUT_URL"`
|
||||
|
||||
// JWT Configuration
|
||||
JWKSCacheExpiry time.Duration `env:"JWKS_CACHE_EXPIRY" envDefault:"1h"`
|
||||
TokenExpiry time.Duration `env:"TOKEN_EXPIRY" envDefault:"24h"`
|
||||
|
||||
// Feature Flags
|
||||
DisableAuth bool `env:"DISABLE_AUTH" envDefault:"false"`
|
||||
}
|
||||
|
||||
// Dynamic URL generation
|
||||
func (c Config) GetAuthorizationURL() string {
|
||||
return fmt.Sprintf("https://%s.auth.%s.amazoncognito.com/oauth2/authorize",
|
||||
c.Domain, extractRegion(c.UserPoolID))
|
||||
}
|
||||
|
||||
func (c Config) GetTokenURL() string {
|
||||
return fmt.Sprintf("https://%s.auth.%s.amazoncognito.com/oauth2/token",
|
||||
c.Domain, extractRegion(c.UserPoolID))
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables Reference
|
||||
|
||||
### Core Application Variables
|
||||
```bash
|
||||
# Logging and Debugging
|
||||
LOG_LEVEL=DEBUG|INFO|WARN|ERROR # Default: INFO
|
||||
ENABLE_OTEL=true|false # Default: false
|
||||
|
||||
# Server Configuration
|
||||
PORT=8080 # Default: 8080
|
||||
HTTP_HOST=0.0.0.0 # Default: 0.0.0.0
|
||||
METRICS_PATH=/metrics # Default: /metrics
|
||||
|
||||
# Feature Flags
|
||||
DISABLE_AUTH=true|false # Default: false (enable for local dev)
|
||||
```
|
||||
|
||||
### Database Variables
|
||||
```bash
|
||||
# Connection
|
||||
PGHOST=localhost # Database host
|
||||
PGPORT=5432 # Database port
|
||||
PGDATABASE=queryorchestration # Database name
|
||||
PGUSER=postgres # Database user
|
||||
PGPASSWORD=password # Database password
|
||||
|
||||
# Connection Pool
|
||||
DB_MAX_OPEN_CONNS=25 # Default: 25
|
||||
DB_MAX_IDLE_CONNS=25 # Default: 25
|
||||
DB_CONN_MAX_LIFETIME=300 # Seconds, Default: 300
|
||||
|
||||
# Security
|
||||
DB_SSL_MODE=require|disable # Default: require
|
||||
DB_NOSSL=true|false # Default: false (for local dev)
|
||||
```
|
||||
|
||||
### AWS Configuration Variables
|
||||
```bash
|
||||
# Credentials
|
||||
AWS_REGION=us-east-1 # AWS region
|
||||
AWS_ACCESS_KEY_ID=your-access-key # AWS access key
|
||||
AWS_SECRET_ACCESS_KEY=your-secret # AWS secret key
|
||||
AWS_SESSION_TOKEN=session-token # Optional session token
|
||||
AWS_PROFILE=default # Optional AWS profile
|
||||
|
||||
# LocalStack/Testing
|
||||
AWS_ENDPOINT_URL=http://localstack:4566 # Override AWS endpoints
|
||||
AWS_S3_USE_PATH_STYLE=true|false # S3 path style (LocalStack)
|
||||
|
||||
# Service Configuration
|
||||
S3_BUCKET=documents-bucket # S3 bucket for documents
|
||||
TEXTRACT_ROLE_ARN=arn:aws:iam::... # Textract service role
|
||||
```
|
||||
|
||||
### Queue Configuration Variables
|
||||
```bash
|
||||
# Queue URLs (SQS)
|
||||
STORE_EVENT_URL=https://sqs.region.amazonaws.com/account/store-events
|
||||
DOCUMENT_INIT_URL=https://sqs.region.amazonaws.com/account/doc-init
|
||||
DOCUMENT_SYNC_URL=https://sqs.region.amazonaws.com/account/doc-sync
|
||||
DOCUMENT_CLEAN_URL=https://sqs.region.amazonaws.com/account/doc-clean
|
||||
DOCUMENT_TEXT_URL=https://sqs.region.amazonaws.com/account/doc-text
|
||||
QUERY_SYNC_URL=https://sqs.region.amazonaws.com/account/query-sync
|
||||
QUERY_URL=https://sqs.region.amazonaws.com/account/query
|
||||
CLIENT_SYNC_URL=https://sqs.region.amazonaws.com/account/client-sync
|
||||
QUERY_VERSION_SYNC_URL=https://sqs.region.amazonaws.com/account/query-version-sync
|
||||
|
||||
# Queue Processing
|
||||
QUEUE_MAX_MESSAGES=10 # Messages per batch
|
||||
QUEUE_WAIT_TIME=20 # Long polling seconds
|
||||
QUEUE_VISIBILITY_TIMEOUT=30 # Message visibility
|
||||
QUEUE_WORKER_COUNT=5 # Concurrent workers
|
||||
QUEUE_MAX_RETRIES=3 # Retry attempts
|
||||
QUEUE_RETRY_DELAY=5 # Retry delay seconds
|
||||
```
|
||||
|
||||
### Authentication Variables
|
||||
```bash
|
||||
# AWS Cognito
|
||||
COGNITO_USER_POOL_ID=us-east-1_xxxxxxx # Cognito User Pool ID
|
||||
COGNITO_CLIENT_ID=client-id # Cognito App Client ID
|
||||
COGNITO_CLIENT_SECRET=client-secret # Cognito App Client Secret
|
||||
COGNITO_DOMAIN=your-domain # Cognito domain name
|
||||
|
||||
# URLs
|
||||
COGNITO_REDIRECT_URL=http://localhost:8080/login-callback
|
||||
COGNITO_LOGOUT_URL=http://localhost:8080/logout
|
||||
|
||||
# JWT Settings
|
||||
JWKS_CACHE_EXPIRY=1h # JWKS cache duration
|
||||
TOKEN_EXPIRY=24h # JWT token expiry
|
||||
```
|
||||
|
||||
## Configuration Validation
|
||||
|
||||
### Required Field Validation
|
||||
```go
|
||||
type Config struct {
|
||||
DatabaseURL string `env:"DATABASE_URL,required"`
|
||||
S3Bucket string `env:"S3_BUCKET,required"`
|
||||
QueueURL string `env:"QUEUE_URL,required"`
|
||||
}
|
||||
|
||||
// Validation on startup
|
||||
func (c *Config) Validate() error {
|
||||
var errors []string
|
||||
|
||||
if c.DatabaseURL == "" {
|
||||
errors = append(errors, "DATABASE_URL is required")
|
||||
}
|
||||
|
||||
if c.S3Bucket == "" {
|
||||
errors = append(errors, "S3_BUCKET is required")
|
||||
}
|
||||
|
||||
if len(errors) > 0 {
|
||||
return fmt.Errorf("configuration validation failed: %s", strings.Join(errors, ", "))
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
### Secret Masking
|
||||
```go
|
||||
// Automatic masking of sensitive values in logs
|
||||
type Config struct {
|
||||
DatabasePassword string `env:"PGPASSWORD" envSecret:"true"`
|
||||
AWSSecretKey string `env:"AWS_SECRET_ACCESS_KEY" envSecret:"true"`
|
||||
JWTSecret string `env:"JWT_SECRET" envSecret:"true"`
|
||||
}
|
||||
|
||||
// String() method automatically masks secrets
|
||||
func (c Config) String() string {
|
||||
return env.String(c) // Automatically masks fields tagged with envSecret
|
||||
}
|
||||
```
|
||||
|
||||
## Development vs Production Configuration
|
||||
|
||||
### Development Environment (.env)
|
||||
```bash
|
||||
# Local development overrides
|
||||
LOG_LEVEL=DEBUG
|
||||
DISABLE_AUTH=true
|
||||
DB_NOSSL=true
|
||||
AWS_ENDPOINT_URL=http://localstack:4566
|
||||
AWS_S3_USE_PATH_STYLE=true
|
||||
|
||||
# Local service URLs
|
||||
PGHOST=localhost
|
||||
PGPORT=5432
|
||||
```
|
||||
|
||||
### Production Environment
|
||||
```bash
|
||||
# Production settings
|
||||
LOG_LEVEL=INFO
|
||||
DISABLE_AUTH=false
|
||||
DB_SSL_MODE=require
|
||||
|
||||
# Production AWS
|
||||
AWS_REGION=us-east-1
|
||||
# Credentials from IAM roles or environment
|
||||
|
||||
# Production database
|
||||
PGHOST=production-db.region.rds.amazonaws.com
|
||||
PGPORT=5432
|
||||
DB_SSL_MODE=require
|
||||
```
|
||||
|
||||
### Testing Environment
|
||||
```bash
|
||||
# Test-specific overrides
|
||||
LOG_LEVEL=WARN
|
||||
DISABLE_AUTH=true
|
||||
DB_NOSSL=true
|
||||
|
||||
# Test containers
|
||||
PGHOST=test-container
|
||||
AWS_ENDPOINT_URL=http://test-localstack:4566
|
||||
```
|
||||
|
||||
## Configuration Best Practices
|
||||
|
||||
### Security
|
||||
- **Never commit secrets**: Use environment variables or secret management
|
||||
- **Mask sensitive logs**: Use `envSecret` tag for automatic masking
|
||||
- **Validate inputs**: Check configuration values at startup
|
||||
- **Use least privilege**: Configure minimal required permissions
|
||||
|
||||
### Maintainability
|
||||
- **Default values**: Provide sensible defaults for optional configuration
|
||||
- **Documentation**: Document all environment variables and their purpose
|
||||
- **Type safety**: Use strong typing and validation
|
||||
- **Composition**: Build configuration from smaller, focused interfaces
|
||||
|
||||
### Testing
|
||||
- **Override-friendly**: Allow easy configuration overrides for tests
|
||||
- **Isolation**: Each test should have independent configuration
|
||||
- **Realistic defaults**: Test configuration should mirror production patterns
|
||||
- **Environment separation**: Clear separation between dev/test/prod configs
|
||||
@@ -0,0 +1,503 @@
|
||||
# Operations Guide
|
||||
|
||||
## Deployment Procedures
|
||||
|
||||
### Local Development Deployment
|
||||
```bash
|
||||
# 1. Start development environment
|
||||
devbox shell
|
||||
|
||||
# 2. Start all services with dependencies
|
||||
task dev:up
|
||||
|
||||
# 3. Verify deployment
|
||||
task health:check
|
||||
|
||||
# 4. View logs
|
||||
task logs:follow
|
||||
```
|
||||
|
||||
### Production Deployment
|
||||
|
||||
#### Prerequisites
|
||||
- Container orchestration platform (Docker Swarm, Kubernetes, ECS)
|
||||
- PostgreSQL 17.2+ database
|
||||
- AWS account with required services
|
||||
- Load balancer for API endpoints
|
||||
- Monitoring infrastructure
|
||||
|
||||
#### Deployment Steps
|
||||
```bash
|
||||
# 1. Build production images
|
||||
task docker:build
|
||||
|
||||
# 2. Push to container registry
|
||||
task docker:push
|
||||
|
||||
# 3. Update configuration
|
||||
kubectl apply -f k8s/config/
|
||||
|
||||
# 4. Deploy services
|
||||
kubectl apply -f k8s/services/
|
||||
|
||||
# 5. Verify deployment
|
||||
kubectl rollout status deployment/query-api
|
||||
```
|
||||
|
||||
#### Environment-Specific Configurations
|
||||
|
||||
**Staging Environment**:
|
||||
```yaml
|
||||
# staging.env
|
||||
LOG_LEVEL=DEBUG
|
||||
ENABLE_OTEL=true
|
||||
DB_SSL_MODE=require
|
||||
|
||||
# Staging AWS resources
|
||||
AWS_REGION=us-west-2
|
||||
S3_BUCKET=doczy-staging-documents
|
||||
```
|
||||
|
||||
**Production Environment**:
|
||||
```yaml
|
||||
# production.env
|
||||
LOG_LEVEL=INFO
|
||||
ENABLE_OTEL=true
|
||||
DB_SSL_MODE=require
|
||||
|
||||
# Production AWS resources
|
||||
AWS_REGION=us-east-1
|
||||
S3_BUCKET=doczy-production-documents
|
||||
```
|
||||
|
||||
### Rolling Updates
|
||||
```bash
|
||||
# Update API service with zero downtime
|
||||
kubectl set image deployment/query-api query-api=registry/query-api:v1.2.0
|
||||
|
||||
# Monitor rollout
|
||||
kubectl rollout status deployment/query-api
|
||||
|
||||
# Rollback if needed
|
||||
kubectl rollout undo deployment/query-api
|
||||
```
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
#### System Metrics
|
||||
All services automatically expose metrics at `/metrics`:
|
||||
- **HTTP requests**: Request count, duration, status codes
|
||||
- **Database connections**: Pool usage, connection count, query duration
|
||||
- **Queue processing**: Message count, processing time, errors
|
||||
- **Business metrics**: Documents processed, queries executed, results generated
|
||||
|
||||
#### Custom Metrics Example
|
||||
```go
|
||||
// Counter for processed documents
|
||||
var documentsProcessed = promauto.NewCounterVec(
|
||||
prometheus.CounterOpts{
|
||||
Name: "documents_processed_total",
|
||||
Help: "Total number of documents processed",
|
||||
},
|
||||
[]string{"client_id", "status"},
|
||||
)
|
||||
|
||||
// Histogram for query execution time
|
||||
var queryExecutionTime = promauto.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "query_execution_duration_seconds",
|
||||
Help: "Time spent executing queries",
|
||||
},
|
||||
[]string{"query_type"},
|
||||
)
|
||||
```
|
||||
|
||||
#### Key Metrics to Monitor
|
||||
```promql
|
||||
# API Health
|
||||
up{job="query-api"}
|
||||
http_requests_total{job="query-api"}
|
||||
http_request_duration_seconds{job="query-api"}
|
||||
|
||||
# Database Health
|
||||
pg_connections_active
|
||||
pg_connections_max_reached
|
||||
query_duration_seconds
|
||||
|
||||
# Queue Health
|
||||
sqs_messages_visible
|
||||
sqs_messages_inflight
|
||||
queue_processing_duration_seconds
|
||||
|
||||
# Business Metrics
|
||||
documents_processed_total
|
||||
query_executions_total
|
||||
export_operations_total
|
||||
```
|
||||
|
||||
### Health Checks
|
||||
|
||||
#### Service Health Endpoints
|
||||
Each service exposes a health endpoint:
|
||||
```bash
|
||||
# API service health
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Individual service health
|
||||
curl http://localhost:8081/health # storeEventRunner
|
||||
curl http://localhost:8082/health # docInitRunner
|
||||
# ... etc
|
||||
```
|
||||
|
||||
#### Health Check Implementation
|
||||
```go
|
||||
func (h *HealthHandler) IsHealthy(ctx echo.Context) error {
|
||||
status := make(map[string]string)
|
||||
|
||||
// Check database connectivity
|
||||
if err := h.db.Ping(ctx.Request().Context()); err != nil {
|
||||
status["database"] = "unhealthy: " + err.Error()
|
||||
return ctx.JSON(http.StatusServiceUnavailable, status)
|
||||
}
|
||||
status["database"] = "healthy"
|
||||
|
||||
// Check queue connectivity
|
||||
if err := h.queue.Health(); err != nil {
|
||||
status["queue"] = "unhealthy: " + err.Error()
|
||||
return ctx.JSON(http.StatusServiceUnavailable, status)
|
||||
}
|
||||
status["queue"] = "healthy"
|
||||
|
||||
return ctx.JSON(http.StatusOK, status)
|
||||
}
|
||||
```
|
||||
|
||||
#### Kubernetes Health Checks
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: query-api
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: query-api
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
```
|
||||
|
||||
### Alerting Rules
|
||||
|
||||
#### Critical Alerts
|
||||
```yaml
|
||||
# Service Down
|
||||
- alert: ServiceDown
|
||||
expr: up == 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Service {{ $labels.instance }} is down"
|
||||
|
||||
# High Error Rate
|
||||
- alert: HighErrorRate
|
||||
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High error rate on {{ $labels.instance }}"
|
||||
|
||||
# Database Connection Issues
|
||||
- alert: DatabaseConnectionHigh
|
||||
expr: pg_connections_active / pg_connections_max > 0.8
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Database connection pool usage high"
|
||||
|
||||
# Queue Backlog
|
||||
- alert: QueueBacklog
|
||||
expr: sqs_messages_visible > 1000
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Queue backlog detected: {{ $value }} messages"
|
||||
```
|
||||
|
||||
#### Business Logic Alerts
|
||||
```yaml
|
||||
# Document Processing Delays
|
||||
- alert: DocumentProcessingDelay
|
||||
expr: increase(documents_uploaded_total[1h]) - increase(documents_processed_total[1h]) > 100
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Document processing falling behind uploads"
|
||||
|
||||
# Query Failures
|
||||
- alert: QueryFailures
|
||||
expr: rate(query_executions_total{status="failed"}[10m]) > 0.05
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High query failure rate detected"
|
||||
```
|
||||
|
||||
## Log Analysis and Debugging
|
||||
|
||||
### Structured Logging Format
|
||||
```json
|
||||
{
|
||||
"time": "2024-01-01T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"msg": "HTTP request",
|
||||
"method": "POST",
|
||||
"uri": "/client",
|
||||
"status": 201,
|
||||
"latency": "45.123ms",
|
||||
"remote_ip": "192.168.1.100",
|
||||
"user_agent": "curl/7.68.0",
|
||||
"trace_id": "abc123def456",
|
||||
"span_id": "789ghi012jkl"
|
||||
}
|
||||
```
|
||||
|
||||
### Log Aggregation Queries
|
||||
|
||||
#### Common Log Searches
|
||||
```bash
|
||||
# Error logs across all services
|
||||
level:"ERROR" | timechart span=5m count by service
|
||||
|
||||
# API endpoint performance
|
||||
service:"queryAPI" AND uri:"/client" | stats avg(latency) by uri
|
||||
|
||||
# Database query performance
|
||||
msg:"database query" | stats avg(duration) by query_type
|
||||
|
||||
# Authentication failures
|
||||
msg:"authentication failed" | stats count by remote_ip
|
||||
|
||||
# Queue processing errors
|
||||
service:"*Runner" AND level:"ERROR" | timechart span=1m count
|
||||
```
|
||||
|
||||
#### Performance Analysis
|
||||
```bash
|
||||
# Slow API requests
|
||||
service:"queryAPI" AND latency > 1000ms | head 100
|
||||
|
||||
# Database connection issues
|
||||
msg:"database connection" AND level:"ERROR"
|
||||
|
||||
# Queue processing bottlenecks
|
||||
service:"queryRunner" | stats avg(processing_time) by query_type
|
||||
```
|
||||
|
||||
### Debugging Common Issues
|
||||
|
||||
#### Service Won't Start
|
||||
```bash
|
||||
# Check configuration
|
||||
env | grep -E "(PGHOST|AWS_|QUEUE_)"
|
||||
|
||||
# Validate database connectivity
|
||||
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE -c "SELECT 1"
|
||||
|
||||
# Test AWS credentials
|
||||
aws sts get-caller-identity
|
||||
|
||||
# Check queue connectivity
|
||||
aws sqs get-queue-attributes --queue-url $QUEUE_URL --attribute-names All
|
||||
```
|
||||
|
||||
#### High Memory Usage
|
||||
```bash
|
||||
# Generate memory profile
|
||||
curl http://localhost:8080/debug/pprof/heap > heap.prof
|
||||
go tool pprof heap.prof
|
||||
|
||||
# Check for goroutine leaks
|
||||
curl http://localhost:8080/debug/pprof/goroutine > goroutine.prof
|
||||
go tool pprof goroutine.prof
|
||||
```
|
||||
|
||||
#### Database Performance Issues
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNqNU9tqg0AQ_ZXBl14woX1tyEObWGpJIiSBUhBku07MEh1TXdtKyL9319RrbKkPi3Nm58yZmZ2DwWMfjTswBgOYbJHvgHEpPhB4TITqN6bUpZU1syZrhWUkL6-v4HHpzGEfeKlk0isChMzh5claWqAxhDFcnIguRi65pNhtH0mKTQ5pGH_Ce4aJwJpa27kJETLy8Au5J0WEJnAWhim41EpYZIgUm_Y4y6m1hIfXTihMrdVEuWf23F7D7U2p4lRjGPNdnftNm-h7BTrcCx_uVxWoTNMlOH0lWJY8zFIkprI1AhSUdCIEBT3kGj1n1-iv9NrZw99UVDSyqafq1l-JqiieJYm6W0d5grxabhJzTNN6HpxJFsbBcP9TYLuXLj079qJzr_1muvrBWZzXpLs2Ph9TL31DRtX2irQehD5lvscz3tLxX-3NNrby9KlvPQSXTguzcLpPMEgYSfRHhglGhEnEhK9W9GDIrRqJXlYfNywLpXE8fgM8B1iJ)
|
||||
|
||||
```sql
|
||||
-- Check active connections
|
||||
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
|
||||
|
||||
-- Identify slow queries
|
||||
SELECT query, mean_exec_time, calls
|
||||
FROM pg_stat_statements
|
||||
ORDER BY mean_exec_time DESC
|
||||
LIMIT 10;
|
||||
|
||||
-- Check locks
|
||||
SELECT blocked_locks.pid AS blocked_pid,
|
||||
blocked_activity.usename AS blocked_user,
|
||||
blocking_locks.pid AS blocking_pid,
|
||||
blocking_activity.usename AS blocking_user,
|
||||
blocked_activity.query AS blocked_statement,
|
||||
blocking_activity.query AS current_statement_in_blocking_process
|
||||
FROM pg_catalog.pg_locks blocked_locks
|
||||
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
|
||||
JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
|
||||
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
|
||||
WHERE NOT blocked_locks.granted;
|
||||
```
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Database Backup
|
||||
```bash
|
||||
# Daily backup script
|
||||
#!/bin/bash
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
BACKUP_FILE="queryorchestration_backup_$DATE.sql"
|
||||
|
||||
pg_dump -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE \
|
||||
--no-password --compress=9 --file=$BACKUP_FILE
|
||||
|
||||
# Upload to S3
|
||||
aws s3 cp $BACKUP_FILE s3://backups-bucket/database/
|
||||
|
||||
# Cleanup old local backups
|
||||
find /backups -name "*.sql" -mtime +7 -delete
|
||||
```
|
||||
|
||||
### S3 Document Backup
|
||||
```bash
|
||||
# Cross-region replication configuration
|
||||
aws s3api put-bucket-replication \
|
||||
--bucket source-bucket \
|
||||
--replication-configuration file://replication.json
|
||||
|
||||
# Point-in-time recovery setup
|
||||
aws s3api put-bucket-versioning \
|
||||
--bucket documents-bucket \
|
||||
--versioning-configuration Status=Enabled
|
||||
```
|
||||
|
||||
### Disaster Recovery
|
||||
|
||||
#### Database Recovery
|
||||
```bash
|
||||
# Restore from backup
|
||||
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE < backup.sql
|
||||
|
||||
# Point-in-time recovery (if WAL archiving enabled)
|
||||
pg_basebackup -h $PGHOST -D /recovery -U postgres -v -P -W
|
||||
```
|
||||
|
||||
#### Service Recovery
|
||||
```bash
|
||||
# Restart all services
|
||||
kubectl rollout restart deployment
|
||||
|
||||
# Verify service health
|
||||
kubectl get pods -l app=query-orchestration
|
||||
|
||||
# Check service connectivity
|
||||
task health:check:all
|
||||
```
|
||||
|
||||
## Capacity Planning
|
||||
|
||||
### Resource Requirements
|
||||
|
||||
#### Production Sizing Guidelines
|
||||
| Component | CPU | Memory | Storage | Network |
|
||||
|-----------|-----|--------|---------|---------|
|
||||
| queryAPI | 2-4 cores | 4-8 GB | 20 GB | 1 Gbps |
|
||||
| Runners (each) | 1-2 cores | 2-4 GB | 10 GB | 500 Mbps |
|
||||
| Database | 4-8 cores | 16-32 GB | 1+ TB SSD | 10 Gbps |
|
||||
| Total System | 16-32 cores | 64-128 GB | 2+ TB | 10+ Gbps |
|
||||
|
||||
#### Scaling Thresholds
|
||||
- **CPU Usage**: Scale up when > 70% for 5 minutes
|
||||
- **Memory Usage**: Scale up when > 80% for 5 minutes
|
||||
- **Queue Depth**: Scale runners when > 100 messages for 10 minutes
|
||||
- **Database Connections**: Alert when > 80% of max connections
|
||||
|
||||
### Performance Baselines
|
||||
```bash
|
||||
# Expected performance targets
|
||||
API Response Time: < 200ms (95th percentile)
|
||||
Document Processing: < 5 minutes per document
|
||||
Query Execution: < 30 seconds per query
|
||||
Database Query Time: < 100ms (average)
|
||||
Queue Processing: < 1 second per message
|
||||
```
|
||||
|
||||
### Growth Planning
|
||||
```bash
|
||||
# Monthly capacity review
|
||||
- Document upload volume trends
|
||||
- Query execution frequency
|
||||
- Storage growth rate
|
||||
- API request patterns
|
||||
- Resource utilization trends
|
||||
```
|
||||
|
||||
## Security Operations
|
||||
|
||||
### Security Monitoring
|
||||
```bash
|
||||
# Authentication failures
|
||||
grep "authentication failed" /var/log/queryapi/*.log | wc -l
|
||||
|
||||
# Unusual API access patterns
|
||||
tail -f /var/log/queryapi/access.log | grep -E "(POST|PUT|DELETE)"
|
||||
|
||||
# Database access monitoring
|
||||
SELECT * FROM pg_stat_activity WHERE state = 'active' AND backend_type = 'client backend';
|
||||
```
|
||||
|
||||
### Incident Response
|
||||
|
||||
#### Security Incident Procedure
|
||||
1. **Detect**: Monitor alerts and logs for anomalies
|
||||
2. **Assess**: Determine scope and impact
|
||||
3. **Contain**: Isolate affected systems
|
||||
4. **Investigate**: Analyze logs and system state
|
||||
5. **Recover**: Restore normal operations
|
||||
6. **Learn**: Document and improve procedures
|
||||
|
||||
#### Emergency Contacts
|
||||
- **Platform Team**: platform-team@company.com
|
||||
- **Security Team**: security@company.com
|
||||
- **Database Team**: dba@company.com
|
||||
- **AWS Support**: Enterprise support case
|
||||
|
||||
### Compliance and Auditing
|
||||
```bash
|
||||
# Audit log collection
|
||||
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=query-orchestration
|
||||
|
||||
# Access log analysis
|
||||
grep -E "(POST|PUT|DELETE)" /var/log/nginx/access.log | awk '{print $1, $7, $9}' | sort | uniq -c
|
||||
|
||||
# Database audit queries
|
||||
SELECT * FROM pg_stat_user_tables WHERE n_tup_ins + n_tup_upd + n_tup_del > 0;
|
||||
```
|
||||
@@ -0,0 +1,188 @@
|
||||
# DoczyAI Query Orchestration Platform Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This documentation provides comprehensive coverage of the DoczyAI Query Orchestration Platform - a cloud-native document processing and query orchestration system designed to extract structured information from documents using configurable, hierarchical queries.
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
### 📋 System Documentation
|
||||
|
||||
1. **[System Overview](01-system-overview.md)**
|
||||
- Platform purpose and business capabilities
|
||||
- Technology stack and architecture overview
|
||||
- High-level component diagram
|
||||
- Deployment architecture and scalability
|
||||
|
||||
2. **[Service Architecture](02-service-architecture.md)**
|
||||
- Detailed service catalog (12 services)
|
||||
- Document processing pipeline flow
|
||||
- Inter-service communication patterns
|
||||
- Queue architecture and message formats
|
||||
|
||||
3. **[API Documentation](03-api-documentation.md)**
|
||||
- REST API endpoints and authentication
|
||||
- Request/response schemas and validation
|
||||
- OpenAPI specification and code generation
|
||||
- Rate limiting and error handling
|
||||
|
||||
4. **[Data Architecture](04-data-architecture.md)**
|
||||
- Database schema and entity relationships
|
||||
- Versioning system and temporal queries
|
||||
- Query dependency management
|
||||
- Performance optimization strategies
|
||||
|
||||
### 🛠️ Developer Documentation
|
||||
|
||||
5. **[Getting Started Guide](05-getting-started.md)**
|
||||
- Prerequisites and local setup
|
||||
- Development environment (Devbox)
|
||||
- Common development workflows
|
||||
- Testing and debugging guidelines
|
||||
|
||||
6. **[Code Organization](06-code-organization.md)**
|
||||
- Project structure and package architecture
|
||||
- Design patterns and naming conventions
|
||||
- Code style guidelines and best practices
|
||||
- Testing patterns and examples
|
||||
|
||||
7. **[Configuration Management](07-configuration-management.md)**
|
||||
- Environment variable hierarchy
|
||||
- Service configuration patterns
|
||||
- Development vs production settings
|
||||
- Security and validation practices
|
||||
|
||||
8. **[Operations Guide](08-operations-guide.md)**
|
||||
- Deployment procedures
|
||||
- Monitoring and alerting setup
|
||||
- Log analysis and debugging
|
||||
- Backup, recovery, and capacity planning
|
||||
|
||||
## Quick Start
|
||||
|
||||
### For New Developers
|
||||
1. Start with **[Getting Started Guide](05-getting-started.md)** for environment setup
|
||||
2. Read **[Code Organization](06-code-organization.md)** to understand the codebase structure
|
||||
3. Review **[Configuration Management](07-configuration-management.md)** for local development
|
||||
4. Explore **[API Documentation](03-api-documentation.md)** for endpoint details
|
||||
|
||||
### For System Architects
|
||||
1. Begin with **[System Overview](01-system-overview.md)** for architectural understanding
|
||||
2. Study **[Service Architecture](02-service-architecture.md)** for microservices design
|
||||
3. Review **[Data Architecture](04-data-architecture.md)** for database design
|
||||
4. Consult **[Operations Guide](08-operations-guide.md)** for deployment planning
|
||||
|
||||
### For Operations Teams
|
||||
1. Focus on **[Operations Guide](08-operations-guide.md)** for deployment and monitoring
|
||||
2. Reference **[Configuration Management](07-configuration-management.md)** for environment setup
|
||||
3. Use **[API Documentation](03-api-documentation.md)** for health checks and troubleshooting
|
||||
4. Understand **[Service Architecture](02-service-architecture.md)** for system dependencies
|
||||
|
||||
## System Characteristics
|
||||
|
||||
### 🏗️ Architecture
|
||||
- **Event-driven microservices** with SQS-based communication
|
||||
- **12 distinct services**: 1 HTTP API + 8 runners + 3 utilities
|
||||
- **Multi-tenant design** with client isolation
|
||||
- **Comprehensive versioning** for queries and configurations
|
||||
|
||||
### 🔧 Technology Stack
|
||||
- **Backend**: Go 1.24.0 with Echo framework
|
||||
- **Database**: PostgreSQL 17.2 with SQLC
|
||||
- **Cloud**: AWS (S3, SQS, Textract, Cognito)
|
||||
- **Containers**: Docker with multi-stage builds
|
||||
- **Monitoring**: Prometheus + OpenTelemetry
|
||||
|
||||
### 📊 Key Capabilities
|
||||
- **Document Processing**: PDF upload, cleaning, and text extraction
|
||||
- **Query Orchestration**: Configurable data extraction with dependencies
|
||||
- **Multi-tenant Support**: Isolated client environments
|
||||
- **Export Operations**: Structured data export in multiple formats
|
||||
- **Version Management**: Full query and configuration versioning
|
||||
|
||||
## Documentation Tools
|
||||
|
||||
### Mermaid Diagram Link Generation
|
||||
|
||||
For programmatic generation of mermaid.live links in documentation, use the provided scripts:
|
||||
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNqFkM1KBDEQBl-lyVydycHbCIKieBEU0VtAmqRnE8wfSUcc1n134zqCl8XrR3dR1F7oZEjMIAa4o0gFmQAhUAnozOTdO8HL0z0sJYW-G4e7ggEW50lFjfy7TNsHfMIkqy4uc5XbNnIaW_FTtSqqOMBDgVYJHle2KXZAIc1-PYXLx7Pz09i8_mBvPzBkT52Nuy5H2iZQotOyheebC7gax8trJf4z7H6Nc-MZLHOus5R_Y0gyjoeMb2mm-J3OvG7K4gzEdtlz7gVbCsewhhZsnsXh8AXqY4CE)
|
||||
|
||||
```bash
|
||||
# Generate a mermaid.live URL from a diagram file
|
||||
cat diagram.mermaid | ./scripts/mermaid-to-url.sh
|
||||
|
||||
# Or use Python directly
|
||||
cat diagram.mermaid | python3 ./scripts/mermaid-to-url.py
|
||||
|
||||
# Example usage
|
||||
echo "graph TD; A-->B" | ./scripts/mermaid-to-url.sh
|
||||
# Output: https://mermaid.live/edit#pako:encoded_diagram
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- **Pako compression**: Uses proper deflate compression for URL optimization
|
||||
- **URL-safe encoding**: Compatible with mermaid.live format
|
||||
- **Pipe-friendly**: Accepts mermaid syntax via stdin
|
||||
- **Cross-platform**: Works with bash or Python 3
|
||||
|
||||
**Use cases:**
|
||||
- Automated documentation generation
|
||||
- CI/CD pipeline diagram embedding
|
||||
- Dynamic diagram URL creation for wikis/docs
|
||||
|
||||
### Post-Processing Documentation with Mermaid Links
|
||||
|
||||
After generating documentation, you can automatically add rendered diagram links to all mermaid diagrams:
|
||||
|
||||
```bash
|
||||
# Add mermaid.live links to all markdown files in a directory
|
||||
python3 ./scripts/add-mermaid-links.py --target-dir ./docs
|
||||
|
||||
# Preview changes without modifying files
|
||||
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --dry-run
|
||||
|
||||
# Process subdirectories recursively
|
||||
python3 ./scripts/add-mermaid-links.py --target-dir ./docs --recursive
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- **Automatic detection**: Finds mermaid diagrams by language tag or content keywords
|
||||
- **Non-destructive**: Preserves all original content, only adds links
|
||||
- **Duplicate prevention**: Skips diagrams that already have links
|
||||
- **Flexible targeting**: Process single directories or recursive scans
|
||||
|
||||
**Workflow integration:**
|
||||
1. Generate your documentation with embedded mermaid diagrams
|
||||
2. Run the link addition script as a post-processing step
|
||||
3. All mermaid diagrams now include clickable links to rendered versions
|
||||
|
||||
## Contributing
|
||||
|
||||
### Development Workflow
|
||||
1. **Setup**: Follow the Getting Started Guide
|
||||
2. **Development**: Use task-based workflows (`task --list`)
|
||||
3. **Testing**: Maintain 80% total, 60% function coverage
|
||||
4. **Documentation**: Update relevant docs with changes
|
||||
|
||||
### Code Standards
|
||||
- **Language**: Go with standard project layout
|
||||
- **Testing**: Table-driven tests with testcontainers
|
||||
- **API**: Contract-first with OpenAPI generation
|
||||
- **Database**: Migration-based schema evolution
|
||||
|
||||
## Support and Resources
|
||||
|
||||
### Internal Resources
|
||||
- **Swagger UI**: http://localhost:8080/swagger/ (when running locally)
|
||||
- **Metrics**: http://localhost:9091 (Prometheus)
|
||||
- **Health Checks**: http://localhost:8080/health
|
||||
|
||||
### External Dependencies
|
||||
- **AWS Services**: S3, SQS, Textract, Cognito
|
||||
- **Database**: PostgreSQL 17.2+
|
||||
- **Monitoring**: Prometheus-compatible metrics
|
||||
|
||||
---
|
||||
|
||||
*This documentation is automatically generated and maintained alongside the codebase. For questions or improvements, please contact the platform team.*
|
||||
@@ -0,0 +1,259 @@
|
||||
# Documentation Generation & Testing
|
||||
|
||||
This directory contains scripts and prompts for automatically generating comprehensive system documentation using Claude AI.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
prompts/
|
||||
├── README.md # This file - overview of the system
|
||||
├── prompt.for.system.documentation.generation.md # Main documentation generation prompt
|
||||
├── regenerate_docs.sh # Production documentation generation script
|
||||
├── test.prompt.md # Test prompt for validating the pipeline
|
||||
└── test_script.sh # Test runner for prompt pipeline validation
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
This documentation generation system uses Claude AI to automatically analyze the codebase and produce comprehensive technical documentation. It follows a two-phase approach:
|
||||
|
||||
1. **Testing Phase**: Validates the pipeline with a simple test
|
||||
2. **Generation Phase**: Produces full system documentation
|
||||
|
||||
## How It Works
|
||||
|
||||
### Documentation Generation Process
|
||||
|
||||
The system uses a comprehensive prompt (`prompt.for.system.documentation.generation.md`) that instructs Claude to:
|
||||
|
||||
1. **Analyze the codebase systematically**:
|
||||
- Survey documentation files and entry points
|
||||
- Map directory structure and major modules
|
||||
- Trace application startup flows
|
||||
- Identify architecture patterns
|
||||
|
||||
2. **Generate structured documentation**:
|
||||
- System overview and architecture
|
||||
- Service architecture and data flows
|
||||
- API documentation and schemas
|
||||
- Developer guides and operational procedures
|
||||
|
||||
3. **Output to temporary location**: All generated docs are placed in `./docs_temp/` for review before replacing existing documentation.
|
||||
|
||||
### Testing Pipeline
|
||||
|
||||
Before running full documentation generation, you can validate the pipeline works correctly:
|
||||
|
||||
```bash
|
||||
# Run the test to ensure Claude integration works
|
||||
./test_script.sh
|
||||
```
|
||||
|
||||
This test:
|
||||
- Creates a temporary docs directory
|
||||
- Pipes a simple test prompt to Claude
|
||||
- Verifies output file generation
|
||||
- Confirms the pipeline is working
|
||||
|
||||
## Usage
|
||||
|
||||
### 1. Test the Pipeline (Recommended First Step)
|
||||
|
||||
```bash
|
||||
cd docs/ai.generated/prompts
|
||||
./test_script.sh
|
||||
```
|
||||
|
||||
**Expected Output**: Creates `../../docs_temp/test_output.md` with test content.
|
||||
|
||||
### 2. Generate Full Documentation
|
||||
|
||||
```bash
|
||||
cd docs/ai.generated/prompts
|
||||
./regenerate_docs.sh <path_to_prompt_file>
|
||||
```
|
||||
|
||||
**Examples**:
|
||||
```bash
|
||||
# Generate full system documentation
|
||||
./regenerate_docs.sh ./prompt.for.system.documentation.generation.md
|
||||
|
||||
# Use relative path from current directory
|
||||
./regenerate_docs.sh prompt.for.system.documentation.generation.md
|
||||
|
||||
# Test with dummy prompt
|
||||
./regenerate_docs.sh test.prompt.md
|
||||
```
|
||||
|
||||
**Process Flow**:
|
||||
1. User provides prompt file path as parameter
|
||||
2. Script validates prompt file exists
|
||||
3. **WARNING**: User must type 'YES' to confirm expensive operation
|
||||
4. Script validates prerequisites (Claude CLI)
|
||||
5. Creates temporary documentation directory (`docs_temp/`)
|
||||
6. Pipes the specified prompt to Claude AI
|
||||
7. Claude analyzes the codebase and generates documentation
|
||||
8. Script prompts user to review and optionally replace existing docs
|
||||
|
||||
### 3. Review and Deploy
|
||||
|
||||
After generation:
|
||||
1. Review generated documentation in `docs_temp/`
|
||||
2. When prompted, choose whether to replace existing docs
|
||||
3. If approved, existing docs are backed up and new ones deployed
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Required Tools
|
||||
|
||||
- **Claude CLI**: Install from https://claude.ai/code
|
||||
- **Bash**: For running the generation scripts
|
||||
- **Project Access**: Must be run from within the project repository
|
||||
|
||||
### Environment Setup
|
||||
|
||||
The scripts automatically:
|
||||
- Detect project root directory
|
||||
- Create necessary temporary directories
|
||||
- Handle backup of existing documentation
|
||||
- Provide colored terminal output for clarity
|
||||
|
||||
## Script Details
|
||||
|
||||
### `regenerate_docs.sh`
|
||||
|
||||
**Purpose**: Main production script for documentation generation
|
||||
|
||||
**Usage**: `./regenerate_docs.sh <path_to_prompt_file>`
|
||||
|
||||
**Features**:
|
||||
- Takes prompt file path as parameter for flexibility
|
||||
- Input validation with helpful usage examples
|
||||
- **WARNING prompt** requiring 'YES' confirmation for expensive operations
|
||||
- Automatic prerequisite checking (Claude CLI, file existence)
|
||||
- Temporary directory management
|
||||
- User confirmation before replacing docs
|
||||
- Automatic backup of existing documentation
|
||||
- Colored output for better UX
|
||||
|
||||
**Safety Features**:
|
||||
- Prompt file existence validation before starting
|
||||
- Warning confirmation (must type exactly 'YES' to proceed)
|
||||
- Backs up existing docs with timestamp
|
||||
- User confirmation before destructive operations
|
||||
- Comprehensive error checking and reporting
|
||||
|
||||
### `test_script.sh`
|
||||
|
||||
**Purpose**: Lightweight test to validate the Claude integration pipeline
|
||||
|
||||
**Features**:
|
||||
- Quick validation of Claude CLI availability
|
||||
- Simple test prompt execution
|
||||
- File generation verification
|
||||
- Pass/fail reporting
|
||||
|
||||
## Generated Documentation Structure
|
||||
|
||||
When successful, the generation process creates:
|
||||
|
||||
```
|
||||
docs_temp/
|
||||
├── 01-system-overview.md # High-level architecture and purpose
|
||||
├── 02-service-architecture.md # Component breakdown and relationships
|
||||
├── 03-api-documentation.md # REST endpoints and schemas
|
||||
├── 04-data-architecture.md # Database schemas and data models
|
||||
├── 05-getting-started.md # Development setup and workflows
|
||||
├── 06-code-organization.md # Package structure and patterns
|
||||
├── 07-configuration-management.md # Config options and environment setup
|
||||
├── 08-operations-guide.md # Deployment and monitoring
|
||||
└── README.md # Documentation index and navigation
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Claude CLI Not Found**
|
||||
```bash
|
||||
Error: Claude CLI not found. Please install it first.
|
||||
```
|
||||
**Solution**: Install Claude CLI from https://claude.ai/code
|
||||
|
||||
2. **No Documentation Generated**
|
||||
```bash
|
||||
Error: No documentation was generated in docs_temp/
|
||||
```
|
||||
**Solutions**:
|
||||
- Check Claude CLI authentication
|
||||
- Verify project structure is accessible
|
||||
- Review Claude's output for error messages
|
||||
|
||||
3. **Permission Issues**
|
||||
```bash
|
||||
Permission denied: ./regenerate_docs.sh
|
||||
```
|
||||
**Solution**: Make scripts executable:
|
||||
```bash
|
||||
chmod +x *.sh
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
1. **Run test script first** to isolate pipeline issues
|
||||
2. **Check Claude output** for specific error messages
|
||||
3. **Verify working directory** is project root
|
||||
4. **Ensure network connectivity** for Claude API access
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Updating the Generation Prompt
|
||||
|
||||
The main prompt (`prompt.for.system.documentation.generation.md`) can be modified to:
|
||||
- Add new documentation sections
|
||||
- Change analysis methodology
|
||||
- Adjust output formatting requirements
|
||||
- Update verification checklist
|
||||
|
||||
### Script Modifications
|
||||
|
||||
Scripts are designed to be maintainable:
|
||||
- Clear variable definitions at the top
|
||||
- Comprehensive error checking
|
||||
- Modular functions for key operations
|
||||
- Extensive comments explaining logic
|
||||
|
||||
## Integration with Development Workflow
|
||||
|
||||
### Recommended Usage Patterns
|
||||
|
||||
1. **After Major Architecture Changes**:
|
||||
```bash
|
||||
./regenerate_docs.sh prompt.for.system.documentation.generation.md
|
||||
```
|
||||
2. **Before Releases**: Ensure documentation matches current implementation
|
||||
3. **Onboarding New Developers**: Fresh, accurate documentation for team members
|
||||
4. **API Changes**: Keep endpoint documentation synchronized with code
|
||||
5. **Testing Pipeline**: Always test with dummy prompt first:
|
||||
```bash
|
||||
./regenerate_docs.sh test.prompt.md
|
||||
```
|
||||
|
||||
### CI/CD Integration Potential
|
||||
|
||||
The scripts could be integrated into automated workflows:
|
||||
- **PR Checks**: Verify docs stay current with code changes
|
||||
- **Release Automation**: Auto-update docs as part of release process
|
||||
- **Scheduled Updates**: Regular documentation refreshes
|
||||
|
||||
## Quality Assurance
|
||||
|
||||
The generation process includes verification steps:
|
||||
- ✅ All major packages documented
|
||||
- ✅ API endpoints match route definitions
|
||||
- ✅ Configuration options are complete
|
||||
- ✅ Database schemas reflect models
|
||||
- ✅ Deployment instructions match infrastructure
|
||||
- ✅ Dependencies and versions are current
|
||||
|
||||
This ensures generated documentation accurately reflects the actual codebase rather than outdated or aspirational descriptions.
|
||||
@@ -0,0 +1,204 @@
|
||||
# **Backend Codebase Analysis & Documentation Generation**
|
||||
|
||||
You are an expert system architect and technical writer tasked with analyzing a large backend codebase and producing comprehensive system and developer documentation. Your analysis should be thorough, accurate, and based entirely on evidence found in the codebase.
|
||||
|
||||
## **Analysis Methodology**
|
||||
|
||||
### **1\. Initial Codebase Survey**
|
||||
|
||||
* Read and catalog all `.md`, `.txt`, and documentation files first
|
||||
* Identify the main entry points (`main.go`, `cmd/` directories)
|
||||
* Map the overall directory structure and identify major modules/packages
|
||||
* Locate configuration files (`.yaml`, `.json`, `.env`, `.toml`, etc.)
|
||||
* Find infrastructure files (`Dockerfile`, `docker-compose.yml`, `Makefile`, CI/CD configs)
|
||||
* If you find mocks used in testing note them but also note that its the project policy going forward to not use mocks in tests.
|
||||
|
||||
### **2\. Architecture Discovery**
|
||||
|
||||
* Trace the application startup flow from main functions.
|
||||
* Identify how many cmd targets there are and what each one does.
|
||||
* Identify and document all major components and their relationships. Which components are shared by which cmd targets.
|
||||
* Map out the service initialization
|
||||
* Document the middleware stack and request/response flow
|
||||
* Identify database connections, external service integrations, and third-party dependencies
|
||||
|
||||
### **3\. Code Structure Analysis**
|
||||
|
||||
* Document all packages and their purposes
|
||||
* Map internal APIs and interfaces between packages
|
||||
* Identify design patterns used (repository, factory, adapter, etc.)
|
||||
* Document error handling strategies and logging patterns
|
||||
* Analyze configuration management and environment variable usage
|
||||
|
||||
### **4\. Operational Analysis**
|
||||
|
||||
* Extract deployment information from Dockerfiles and deployment configs
|
||||
* Document environment variables and configuration options
|
||||
* Identify monitoring, logging, and observability implementations
|
||||
* Map out database schemas and migration strategies
|
||||
* Document API endpoints, authentication, and authorization mechanisms
|
||||
|
||||
## **Required Documentation Outputs**
|
||||
|
||||
### **System Documentation**
|
||||
|
||||
#### **1\. System Overview**
|
||||
|
||||
* High-level architecture diagram (textual description) and mermaid diagrams.
|
||||
* Core business domain and purpose
|
||||
* Technology stack and major dependencies
|
||||
* Deployment architecture and infrastructure requirements
|
||||
|
||||
#### **2\. Service Architecture**
|
||||
|
||||
* Component breakdown with responsibilities
|
||||
* Inter-service communication patterns
|
||||
* Data flow diagrams
|
||||
* Integration points with external systems
|
||||
* Scalability and performance considerations
|
||||
|
||||
#### **3\. API Documentation**
|
||||
|
||||
* All REST/GraphQL/gRPC endpoints
|
||||
* Identify how rest api documentation is generated from the api definitions
|
||||
* Identity and document Request/response schemas
|
||||
* Authentication and authorization requirements
|
||||
* Rate limiting and security considerations
|
||||
* Error response formats and codes
|
||||
|
||||
#### **4\. Data Architecture**
|
||||
|
||||
* Database schemas and relationships
|
||||
* Data models and entity relationships
|
||||
* Migration strategies and versioning
|
||||
* Data persistence patterns
|
||||
* Caching strategies
|
||||
|
||||
### **Developer Documentation**
|
||||
|
||||
#### **1\. Getting Started Guide**
|
||||
|
||||
* Prerequisites and local development setup
|
||||
* Build and run instructions
|
||||
* Testing procedures
|
||||
* Common development workflows
|
||||
|
||||
#### **2\. Code Organization**
|
||||
|
||||
* Package structure and naming conventions
|
||||
* Architectural patterns and design principles
|
||||
* Code style and formatting standards
|
||||
* Dependency management practices
|
||||
|
||||
#### **3\. Configuration Management**
|
||||
|
||||
* All configuration options with descriptions and default values
|
||||
* Environment-specific configurations
|
||||
* Feature flags and toggles
|
||||
* Secret management patterns
|
||||
|
||||
#### **4\. Operations Guide**
|
||||
|
||||
* Deployment procedures
|
||||
* Monitoring and alerting setup
|
||||
* Log analysis and debugging
|
||||
*
|
||||
|
||||
## **Analysis Guidelines**
|
||||
|
||||
### **Code Reading Strategy**
|
||||
|
||||
1. **Start with the entry points** \- Understand how the application bootstraps
|
||||
2. **Follow the request flow** \- Trace a typical request from entry to exit
|
||||
3. **Identify the core domain** \- Understand the business logic and data models
|
||||
4. **Map the boundaries** \- Document package interfaces and responsibilities
|
||||
5. **Extract patterns** \- Identify recurring architectural and design patterns
|
||||
|
||||
### **Evidence-Based Documentation**
|
||||
|
||||
* Quote actual code snippets to support architectural claims
|
||||
* Reference specific files and line numbers when relevant
|
||||
* Validate assumptions by cross-referencing multiple source files
|
||||
* Distinguish between implemented features and TODO comments
|
||||
* Note any discrepancies between existing documentation and actual code
|
||||
|
||||
### **Technical Accuracy Requirements**
|
||||
|
||||
* Verify all API endpoints by examining route definitions
|
||||
* Confirm database schemas by reading migration files or model definitions
|
||||
* Validate configuration options by tracing their usage in code
|
||||
* Cross-reference Docker configurations with application requirements
|
||||
* Ensure all external dependencies are documented with versions
|
||||
|
||||
## **Output Format Requirements**
|
||||
|
||||
* Use clear, hierarchical structure with consistent heading levels
|
||||
* Include code examples and configuration snippets where helpful but keep these to a minimum.
|
||||
* Provide file paths and specific references for all claims
|
||||
* Create logical groupings of related information
|
||||
* Maintain technical accuracy while ensuring readability for different audiences
|
||||
* Organize the docs in levels so that at the highest level you see the big picture items then the reader can drill down as needed where possible.
|
||||
|
||||
## **Verification Checklist**
|
||||
|
||||
Before finalizing documentation, ensure:
|
||||
|
||||
* \[ \] All major packages and their purposes are documented
|
||||
* \[ \] API endpoints match actual route definitions in code
|
||||
* \[ \] Configuration options are complete and accurate
|
||||
* \[ \] Database schemas reflect actual model definitions
|
||||
* \[ \] Deployment instructions match Dockerfile and infrastructure configs
|
||||
* \[ \] Dependencies and versions are current and complete
|
||||
* \[ \] Code examples compile and reflect actual usage patterns
|
||||
|
||||
## **Important Notes**
|
||||
|
||||
* Do not make assumptions about functionality not evident in the code
|
||||
* When existing documentation conflicts with code, flag these discrepancies
|
||||
* Focus on what the system actually does, not what it was intended to do
|
||||
* Include both happy path and error handling scenarios
|
||||
* Document any deprecated or legacy patterns you discover
|
||||
|
||||
Begin your analysis by reading the codebase systematically and then produce the comprehensive documentation as outlined above. Try to organize so that there is a minimal amount (or none if possible) of repeated information in the docs to keep them no longer than they need to be.
|
||||
|
||||
All output should be placed in the ./docs\_temp directory in the form of markdown and mermaid files, etc. for the team to review.
|
||||
|
||||
## Optional Memory Bank Instructions
|
||||
|
||||
You have access to a file-based memory system to track tasks, findings, and progress when working with large codebases or complex projects. If you don’t already have some sort of sequential reasoning capability then you may use this technique to help store temporary working state.
|
||||
|
||||
\#\#\# Setup and Usage:
|
||||
1\. \*\*Directory Structure\*\*: Create and use \`./memory/\` as your working memory directory
|
||||
2\. \*\*File Naming\*\*: Use descriptive \`.md\` filenames like \`tasks.md\`, \`bugs-found.md\`, \`analysis-progress.md\`, \`files-to-review.md\`
|
||||
3\. \*\*List Format\*\*: Use markdown checkboxes for trackable items:
|
||||
\`\`\`markdown
|
||||
\# Task List
|
||||
\- \[ \] Review authentication module
|
||||
\- \[x\] Check database connections
|
||||
\- \[ \] Analyze error handling patterns
|
||||
|
||||
### **When to Use Memory Files:**
|
||||
|
||||
* When analyzing codebases with 50+ files
|
||||
* Tracking multiple bugs or issues across sessions
|
||||
* Managing step-by-step analysis plans
|
||||
* Storing intermediate findings that might be lost
|
||||
* Breaking down large tasks into smaller checkable items
|
||||
|
||||
### **Memory Management:**
|
||||
|
||||
* **Read** existing memory files before starting work
|
||||
* **Update** progress by checking off completed items `- [x]`
|
||||
* **Add** new items as you discover them `- [ ]`
|
||||
* **Create** new memory files for different categories as needed
|
||||
* **Clean up** completed files when projects finish
|
||||
|
||||
### **Example Files:**
|
||||
|
||||
* `current-analysis.md` \- What you're currently working on
|
||||
* `issues-found.md` \- Bugs and problems discovered
|
||||
* `questions.md` \- Things to ask the user about
|
||||
* `next-steps.md` \- Planned future actions
|
||||
|
||||
Always check for existing memory files first which might indicate that you have already begun work on this and the work was interrupted. Then update them as you work.
|
||||
|
||||
Executable
+111
@@ -0,0 +1,111 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Script to regenerate system documentation using Claude
|
||||
# Usage: ./regenerate_docs.sh <path_to_prompt_file>
|
||||
# This script pipes the documentation generation prompt to Claude CLI
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Check if prompt file path is provided
|
||||
if [ $# -eq 0 ]; then
|
||||
echo -e "${RED}Error: Please provide the path to the prompt file${NC}"
|
||||
echo "Usage: $0 <path_to_prompt_file>"
|
||||
echo "Example: $0 ./prompt.for.system.documentation.generation.md"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PROMPT_FILE="$1"
|
||||
|
||||
# Check if prompt file exists
|
||||
if [ ! -f "$PROMPT_FILE" ]; then
|
||||
echo -e "${RED}Error: Prompt file not found at $PROMPT_FILE${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Convert to absolute path for clarity
|
||||
PROMPT_FILE="$(realpath "$PROMPT_FILE")"
|
||||
|
||||
# Warning prompt
|
||||
echo -e "${RED}WARNING: Regeneration of the entire system docs is a time consuming and expensive operation.${NC}"
|
||||
echo -e "${YELLOW}Enter 'YES' to continue. Or press Enter to abort:${NC}"
|
||||
read -r confirmation
|
||||
|
||||
if [ "$confirmation" != "YES" ]; then
|
||||
echo -e "${YELLOW}Operation aborted.${NC}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Get the directory where this script is located
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../../" && pwd)"
|
||||
TEMP_DOCS_DIR="$PROJECT_ROOT/docs_temp"
|
||||
TARGET_DOCS_DIR="$PROJECT_ROOT/docs/ai.generated"
|
||||
|
||||
echo -e "${YELLOW}Starting documentation regeneration...${NC}"
|
||||
echo -e "${GREEN}Using prompt file: $PROMPT_FILE${NC}"
|
||||
|
||||
# Check if claude CLI is available
|
||||
if ! command -v claude &> /dev/null; then
|
||||
echo -e "${RED}Error: Claude CLI not found. Please install it first.${NC}"
|
||||
echo "See: https://claude.ai/code"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Change to project root directory
|
||||
cd "$PROJECT_ROOT"
|
||||
echo -e "${GREEN}Working from project root: $PROJECT_ROOT${NC}"
|
||||
|
||||
# Remove existing temp docs directory if it exists
|
||||
if [ -d "$TEMP_DOCS_DIR" ]; then
|
||||
echo -e "${YELLOW}Removing existing temporary docs directory...${NC}"
|
||||
rm -rf "$TEMP_DOCS_DIR"
|
||||
fi
|
||||
|
||||
# Create temporary docs directory
|
||||
mkdir -p "$TEMP_DOCS_DIR"
|
||||
|
||||
echo -e "${YELLOW}Piping prompt to Claude CLI...${NC}"
|
||||
echo -e "${YELLOW}This may take several minutes depending on codebase size...${NC}"
|
||||
|
||||
# Pipe the prompt to Claude CLI
|
||||
cat "$PROMPT_FILE" | claude
|
||||
|
||||
# Check if documentation was generated
|
||||
if [ ! -d "$TEMP_DOCS_DIR" ] || [ -z "$(ls -A "$TEMP_DOCS_DIR" 2>/dev/null)" ]; then
|
||||
echo -e "${RED}Error: No documentation was generated in $TEMP_DOCS_DIR${NC}"
|
||||
echo -e "${YELLOW}Please check Claude's output above for any errors.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}Documentation generated successfully in $TEMP_DOCS_DIR${NC}"
|
||||
|
||||
# Ask user if they want to move the docs to the target location
|
||||
echo -e "${YELLOW}Would you like to replace the existing docs with the newly generated ones? (y/N)${NC}"
|
||||
read -r response
|
||||
|
||||
if [[ "$response" =~ ^[Yy]$ ]]; then
|
||||
# Backup existing docs if they exist
|
||||
if [ -d "$TARGET_DOCS_DIR" ]; then
|
||||
BACKUP_DIR="${TARGET_DOCS_DIR}.backup.$(date +%Y%m%d_%H%M%S)"
|
||||
echo -e "${YELLOW}Backing up existing docs to $BACKUP_DIR${NC}"
|
||||
mv "$TARGET_DOCS_DIR" "$BACKUP_DIR"
|
||||
fi
|
||||
|
||||
# Move new docs to target location
|
||||
echo -e "${YELLOW}Moving new docs to $TARGET_DOCS_DIR${NC}"
|
||||
mv "$TEMP_DOCS_DIR" "$TARGET_DOCS_DIR"
|
||||
|
||||
echo -e "${GREEN}Documentation update complete!${NC}"
|
||||
echo -e "${GREEN}New docs are now available in: $TARGET_DOCS_DIR${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}New documentation remains in: $TEMP_DOCS_DIR${NC}"
|
||||
echo -e "${YELLOW}You can manually review and move them when ready.${NC}"
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}Documentation regeneration process completed.${NC}"
|
||||
@@ -0,0 +1,24 @@
|
||||
# Test Prompt for Documentation Script
|
||||
|
||||
This is a simple test prompt to verify that the documentation regeneration script works correctly.
|
||||
|
||||
## Instructions
|
||||
|
||||
Please respond with a simple confirmation message and create a single test file in the `./docs_temp` directory to verify the script's file handling logic.
|
||||
|
||||
## Expected Output
|
||||
|
||||
1. Confirm that you received this prompt correctly
|
||||
2. Create a file called `test_output.md` in `./docs_temp` with the following content:
|
||||
|
||||
```markdown
|
||||
# Test Documentation Output
|
||||
|
||||
This is a test file generated by Claude to verify the documentation regeneration script is working correctly.
|
||||
|
||||
Script test: PASSED
|
||||
Timestamp: [current date/time]
|
||||
```
|
||||
|
||||
That's it! Keep the response simple and just create the one test file to validate the script workflow.
|
||||
If the output file is already present then overwrite it.
|
||||
+191
@@ -0,0 +1,191 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import sys
|
||||
import os
|
||||
import argparse
|
||||
import re
|
||||
import json
|
||||
import base64
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
def pako_deflate(data):
|
||||
"""Compress data using the same parameters as pako.deflate"""
|
||||
compress = zlib.compressobj(9, zlib.DEFLATED, 15, 8, zlib.Z_DEFAULT_STRATEGY)
|
||||
compressed_data = compress.compress(data)
|
||||
compressed_data += compress.flush()
|
||||
return compressed_data
|
||||
|
||||
def generate_mermaid_url(mermaid_code):
|
||||
"""Generate a mermaid.live URL from mermaid code"""
|
||||
# Create the mermaid object structure expected by mermaid.live
|
||||
mermaid_obj = {
|
||||
"code": mermaid_code.strip(),
|
||||
"mermaid": {
|
||||
"theme": "default"
|
||||
}
|
||||
}
|
||||
|
||||
# Convert to JSON
|
||||
mermaid_json = json.dumps(mermaid_obj)
|
||||
|
||||
# Compress using pako deflate parameters
|
||||
byte_data = mermaid_json.encode('ascii')
|
||||
compressed = pako_deflate(byte_data)
|
||||
|
||||
# Base64 encode (URL-safe)
|
||||
encoded = base64.b64encode(compressed).decode('ascii')
|
||||
encoded = encoded.replace('+', '-').replace('/', '_')
|
||||
|
||||
return f"https://mermaid.live/edit#pako:{encoded}"
|
||||
|
||||
def process_markdown_file(file_path, dry_run=False):
|
||||
"""Process a single markdown file to add mermaid.live links"""
|
||||
try:
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
except Exception as e:
|
||||
print(f"Error reading {file_path}: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
# Pattern to match mermaid code blocks
|
||||
# First find all code blocks, then check if they contain mermaid
|
||||
code_block_pattern = r'```(\w*)\n(.*?)\n```'
|
||||
|
||||
# Find all code blocks and check if they're mermaid
|
||||
matches = list(re.finditer(code_block_pattern, content, re.MULTILINE | re.DOTALL))
|
||||
|
||||
# Filter for mermaid blocks
|
||||
mermaid_matches = []
|
||||
mermaid_keywords = ['graph', 'flowchart', 'sequenceDiagram', 'classDiagram', 'stateDiagram',
|
||||
'erDiagram', 'journey', 'gantt', 'pie', 'gitgraph', 'mindmap',
|
||||
'timeline', 'quadrantChart', 'xyChart', 'block', 'architecture']
|
||||
|
||||
for match in matches:
|
||||
language = match.group(1).lower() # Language specifier after ```
|
||||
code_content = match.group(2) # Code content
|
||||
|
||||
# Check if it's a mermaid block
|
||||
is_mermaid = (language == 'mermaid' or
|
||||
any(keyword in code_content for keyword in mermaid_keywords))
|
||||
|
||||
if is_mermaid:
|
||||
mermaid_matches.append(match)
|
||||
|
||||
if not mermaid_matches:
|
||||
return False
|
||||
|
||||
# Process matches in reverse order to maintain correct positions
|
||||
modified_content = content
|
||||
changes_made = False
|
||||
|
||||
for match in reversed(mermaid_matches):
|
||||
language = match.group(1) # Language specifier
|
||||
mermaid_code = match.group(2) # The actual mermaid code
|
||||
|
||||
# Check if there's already a link before this mermaid block
|
||||
text_before = modified_content[:match.start()]
|
||||
link_pattern = r'\[link to rendered version here\]\(https://mermaid\.live/[^)]+\)\s*$'
|
||||
|
||||
if re.search(link_pattern, text_before):
|
||||
continue # Skip if link already exists
|
||||
|
||||
try:
|
||||
# Generate the mermaid.live URL
|
||||
mermaid_url = generate_mermaid_url(mermaid_code)
|
||||
|
||||
# Create the link text
|
||||
link_text = f"[link to rendered version here]({mermaid_url})\n\n"
|
||||
|
||||
# Insert the link before the mermaid block
|
||||
before = modified_content[:match.start()]
|
||||
after = modified_content[match.start():]
|
||||
modified_content = before + link_text + after
|
||||
changes_made = True
|
||||
|
||||
print(f"Added link for mermaid diagram in {file_path}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error generating URL for mermaid diagram in {file_path}: {e}", file=sys.stderr)
|
||||
continue
|
||||
|
||||
# Write the modified content back to the file
|
||||
if changes_made and not dry_run:
|
||||
try:
|
||||
with open(file_path, 'w', encoding='utf-8') as f:
|
||||
f.write(modified_content)
|
||||
print(f"Updated {file_path}")
|
||||
except Exception as e:
|
||||
print(f"Error writing {file_path}: {e}", file=sys.stderr)
|
||||
return False
|
||||
elif changes_made and dry_run:
|
||||
print(f"[DRY RUN] Would update {file_path}")
|
||||
|
||||
return changes_made
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Add mermaid.live links to mermaid diagrams in markdown files",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python3 add-mermaid-links.py --target-dir ./docs
|
||||
python3 add-mermaid-links.py --target-dir ./docs --dry-run
|
||||
python3 add-mermaid-links.py --target-dir ./docs --recursive
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--target-dir',
|
||||
required=True,
|
||||
help='Directory to scan for markdown files'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--recursive', '-r',
|
||||
action='store_true',
|
||||
help='Recursively scan subdirectories'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--dry-run',
|
||||
action='store_true',
|
||||
help='Show what would be changed without modifying files'
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
target_dir = Path(args.target_dir)
|
||||
|
||||
if not target_dir.exists():
|
||||
print(f"Error: Directory {target_dir} does not exist", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if not target_dir.is_dir():
|
||||
print(f"Error: {target_dir} is not a directory", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Find all markdown files
|
||||
if args.recursive:
|
||||
md_files = list(target_dir.rglob("*.md"))
|
||||
else:
|
||||
md_files = list(target_dir.glob("*.md"))
|
||||
|
||||
if not md_files:
|
||||
print(f"No markdown files found in {target_dir}")
|
||||
sys.exit(0)
|
||||
|
||||
print(f"Found {len(md_files)} markdown files")
|
||||
if args.dry_run:
|
||||
print("DRY RUN MODE - No files will be modified")
|
||||
|
||||
# Process each file
|
||||
files_modified = 0
|
||||
for md_file in md_files:
|
||||
if process_markdown_file(md_file, args.dry_run):
|
||||
files_modified += 1
|
||||
|
||||
print(f"Processed {len(md_files)} files, {files_modified} files {'would be ' if args.dry_run else ''}modified")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import sys
|
||||
import json
|
||||
import base64
|
||||
import zlib
|
||||
|
||||
def pako_deflate(data):
|
||||
"""Compress data using the same parameters as pako.deflate"""
|
||||
compress = zlib.compressobj(9, zlib.DEFLATED, 15, 8, zlib.Z_DEFAULT_STRATEGY)
|
||||
compressed_data = compress.compress(data)
|
||||
compressed_data += compress.flush()
|
||||
return compressed_data
|
||||
|
||||
def main():
|
||||
# Read all input from stdin
|
||||
mermaid_text = sys.stdin.read().strip()
|
||||
|
||||
if not mermaid_text:
|
||||
print("Error: No mermaid text provided via stdin", file=sys.stderr)
|
||||
print("Usage: cat mermaid.md | python3 mermaid-to-url.py", file=sys.stderr)
|
||||
print(" or: echo 'graph TD; A-->B' | python3 mermaid-to-url.py", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Create the mermaid object structure expected by mermaid.live
|
||||
mermaid_obj = {
|
||||
"code": mermaid_text,
|
||||
"mermaid": {
|
||||
"theme": "default"
|
||||
}
|
||||
}
|
||||
|
||||
# Convert to JSON
|
||||
mermaid_json = json.dumps(mermaid_obj)
|
||||
|
||||
# Compress using pako deflate parameters
|
||||
byte_data = mermaid_json.encode('ascii')
|
||||
compressed = pako_deflate(byte_data)
|
||||
|
||||
# Base64 encode (URL-safe)
|
||||
encoded = base64.b64encode(compressed).decode('ascii')
|
||||
encoded = encoded.replace('+', '-').replace('/', '_')
|
||||
|
||||
# Output the mermaid.live URL
|
||||
print(f"https://mermaid.live/edit#pako:{encoded}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -87,6 +87,10 @@ tasks:
|
||||
run: once
|
||||
cmds:
|
||||
- gomarkdoc --template-file file=docs/templates/root.gotxt ./cmd/...
|
||||
docs:ai:generate:
|
||||
desc: "Generate full system docs ($$)"
|
||||
cmds:
|
||||
- bash docs/ai.generated/prompts/regenerate_docs.sh docs/ai.generated/prompts/prompt.for.system.documentation.generation.md
|
||||
aws:login: aws sso login
|
||||
aws:textract:credentials:
|
||||
- |
|
||||
|
||||
Reference in New Issue
Block a user